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Executive Summary 


This report presents the results of a descriptive study of the Researcher—Practitioner Partnerships 
in Education Research program, a two-year grant program funded by the Institute of Education 
Sciences at the U.S. Department of Education that supports exploratory research within a 
partnership context. In these partnerships, researchers collaborate with practitioners from at least 
one state or local education agency on a research project to investigate a problem of practice and 
to identify strategies to address the key issues. The National Center for Research in Policy and 
Practice (NCRPP), a center funded by the Institute of Education Sciences, conducted the study. 
The study analyzed the goals, activities, and perceived progress of funded partnerships as well as 
participants’ perceptions of the program. 


The study focused on the first three cohorts of researcher-practitioner partnerships (RPPs) 
using a mixed-method, cross-case design. It relied on three sources of data: surveys, interviews, 
and original grant applications. NCRPP researchers developed two survey instruments—one 

for researchers and one for practitioners. Surveys included five previously tested scales of items 
from NCRPP’s national survey of educational leaders’ research use as well as new items that were 
tested and revised through a cognitive piloting process. The study team also developed and pilot- 
tested interview protocols for each group, as well as conducted a systematic document review of 
grant applications. The reporting here focuses on broad themes and patterns that emerged across 
partnerships, rather than on reports of individual partnerships. 


Participants in the Partnerships 

The population consisted of key researchers and practitioners who were involved in the 28 
RPPs initially funded by IES between 2013 and 2015. We created a sample from this population 
for our study by first approaching the principal investigators (PIs) of all grants and inviting 
them to participate in the study. We presented each of them with the names of all researchers 
and practitioners that appeared in the study abstract. We asked them to verify their current 
participation and to nominate other active members of the partnership. The 160 individuals 
confirmed or named by PIs constituted our initial sample roster. 


A total of 106 participants from 27 of the 28 funded RPPs completed the survey, the interview, or 
both. We collected data from at least one researcher and one practitioner in 25 of the RPPs. We 
received survey responses from 104 participants (response rate = 65%) and conducted interviews 
with 98 participants (response rate = 61%). Eight participants completed the survey but not the 
interview; two participants completed the interview but not the survey. 


Researchers and practitioners were both experienced in their roles and held advanced degrees. 
On average, researchers had served almost 11 years and practitioners had served over eight years 
in their current roles. Almost all of the researchers (92%) and almost half of the practitioners 
(44%) held doctoral degrees. 


The 61 participating researchers held the following positions in their own institutions: thirty-four 
(56%) were professors; 19 (31%) were researchers with titles like research scientist; and equal 
numbers (4 researchers each) were center directors (7%) or doctoral students (7%). 
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Over half (59%) of the researchers held senior-level positions as associate or full professors, senior 
researchers, or center directors. 


The 45 participating practitioners represented the following areas of practice: Nineteen (42%) 
were in P-20 research, assessment, and accountability; eight (18%) worked in postsecondary 
policy and practice; equal numbers (five practitioners each) worked in K-12 special education 
(11%), K-12 federal programs (11%), or early childhood policy and practice (11%); two 

(4%) were in K-12 educator evaluation; and one practitioner (2%) worked as a K-12 deputy 
superintendent or chief officer. Over two-thirds (67%) of practitioners were serving in senior- 
level positions (e.g., director of an organization or department). 


Chi-square tests indicated that the proportions of rostered RPP participants in each position were 
similar to proportions among participating researchers (p = 0.796) and participating practitioners 
(p = 0.525). Likewise, the proportions of participating researchers and practitioners in senior- 
level positions mirrored those in the full roster (p = 0.922 and 0.998, respectively). 


Participants in each of the RPPs had worked together in some capacity prior to the IES grant 
beyond planning for their grant application, and 10 of the RPPs had a formal partnership 
beforehand. 


Goals of the Partnerships 

The most commonly reported goals of the RPPs were ones emphasized within the request 

for applications (RFA) for the program: conducting and using research, and impacting local 
improvement efforts. RPPs also pursued the goals of cultivating partnership relationships, 
increasing the capacity of researchers and practitioners to engage in partnership work, and 
informing the work of others through the sharing of strategies and tools with people outside the 
partnership. 


Partnerships reported the greatest progress in their initial work toward building relationships 
and refining their understandings of problems the partnership could address. They reported 
somewhat less progress for longer-term goals such as improving organizational policies and 
processes in the educational organization and impacting student outcomes. 


Conducting and Using Research in the Partnerships 

Research questions, methods, and data sources. As outlined in their proposals, the majority 

of partnerships focused on descriptive or exploratory questions, although some focused on 
investigating causal relationships or validating measures or constructs. Most proposed to use 
mixed methods and to draw on both existing and new data sources. To carry out these research 
plans, partnerships created multi-sector datasets, engaged in secondary analyses of student-, 
teacher-, and school-level data, conducted interviews and focus groups, created case study 
reports, and more. 


Educational leaders’ use of research. Overall, RPP practitioners reported using research in multiple 
ways, including to make decisions (instrumental use), to inform how they thought about issues 
(conceptual use), and to persuade others of a particular point of view (symbolic use). In the 
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survey, of those practitioners who participated in directing resources to a program or redesigning 
a program, over three-quarters reported using research instrumentally in those activities. Over 
three-quarters of RPP participants also reported that research had frequently or all of the time 
“expanded (their) understanding of an issue,’ a conceptual use of research. Almost two-thirds of 
participants reported using research to “mobilize support for an important issue” frequently or all 
of the time, a form of symbolic use. 


Activities of the Partnerships 

All partnerships were engaged in problem refinement and analysis of data, and nearly half were 
doing some design work together that involved developing programs and practices that they were 
testing or planning to test. More than one-third were revising interventions they had developed 
together, an activity that might be expected in an IES Development and Innovation grant. 


Communication Within the Partnerships 
Participants in the RPPs maintained regular and frequent communication across partners, with 
most members engaging in weekly emails or phone calls. 


Challenges Within the Partnerships 

Roughly half of the partnerships identified organizational turnover and difficulty obtaining 
usable data as key challenges to their work together. Other leading challenges were synchronizing 
schedules of researchers and practitioners and accommodating different timelines for getting 
work done. 


Perceptions the Partnerships 

Researchers and practitioners alike highly valued their participation in partnership work, 

with 100% of those surveyed either agreeing or strongly agreeing that they would participate 

in another RPP in the future. Practitioners named several benefits, including helping to shift 
organizational culture for research use and increasing access to resources and expertise aimed at 
understanding and addressing specific problems of practice. Both researchers and practitioners 
also felt that the quality and applicability of research increased as a result of the partnership. 


Planned Future Activities of the Partnerships 

Five RPPs indicated plans to apply for new grants, 10 were in the midst of applying, and five had 
secured additional funding at the time of the interview. In spite of shared interest in continuing, a 
number of participants noted that the future of their partnerships was contingent upon securing 
additional funding. 


Guidance for IES 

Half of participants believed that a two-year time frame was adequate for RPPs to achieve their 
goals. Many suggested that new RPPs could benefit from a three-year grant period to accomplish 
their aims; however, a longer grant period would require a proportionate increase in funding. 
Participants were similarly concerned about the sustainability of their partnerships. One common 
suggestion was to create a supplemental funding opportunity that would support an additional 
year or two of project work for partnerships that had been successful in meeting their initial 
goals. 
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Researchers and practitioners requested opportunities to connect with other IES-funded RPPs, 
especially if the meetings focused on common challenges of working in partnerships. They also 
encouraged a wide range of dissemination or communications activities beyond peer-reviewed 
journal articles. Finally, some participants suggested a quicker turnaround for application review 
to better align with practitioners’ pace of work. 
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Introduction 


There is growing interest in research-practice partnerships (RPPs)—collaborations between 
researchers and practitioners formed to investigate problems of practice and solutions for 
improving educational outcomes.! Advocates argue that such partnerships enable greater 
research use in decision making * and address persistent issues in education.* Funders share 
enthusiasm for the approach and are investing considerable resources to develop new RPPs, 
including the Institute for Education Sciences, the National Science Foundation, and the 
Spencer Foundation. Research in fields outside of education (e.g., public health, mental health, 
criminology) provides evidence of the promise of RPPs in improving individual outcomes.* 


Despite large investments from funders to develop and support RPPs in education, there has 
been little systematic research on them.* Available literature consists mainly of retrospective, 
first-person accounts from researchers, focused on challenges they experienced creating and 
sustaining partnerships.° Further, the small body of existing empirical research consists primarily 
of single case studies.’ We do not yet have a good sense of the range of ways researchers and 
practitioners structure their work together, the challenges that emerge, the value of such 
approaches, or the degree to which participation in partnerships fosters research use among 
educational decision makers. There is a need especially for comparative studies of RPPs to better 
understand how partnerships vary in their goals and activities. 


To address this need, this report presents initial findings from the first phase of a study of 
researchers and practitioners engaged in RPPs funded by the Institute of Education Sciences 
(IES) at the U.S. Department of Education. The study was conducted by the IES-funded National 
Center for Research in Policy and Practice (NCRPP), a research and development center focused 
on the study of research use among educational leaders in the United States. 


In 2013, IES launched the Researcher-Practitioner Partnerships in Education Research program, 
which we refer to in this report as the RPP program. Under the leadership of then-IES director 
John Easton, this program was part of a focus on partnerships as a strategy to support rigorous, 
relevant research that meets the needs of local communities and builds local capacity. The two- 
year grant program supports exploratory research within a partnership context. Researchers 

and practitioners collaborate on a research project to explore a problem of practice, identify 
strategies to address the key issues, and engage in preliminary design work related to the problem 
of practice. The formal requirement for an RPP grant is, at a minimum, a partnership between 

a research institution and a state or local education agency. The purpose of this program was 
described in the IES request for applications (RFA) for awards beginning in fiscal year 2017: 


The Researcher-Practitioner Partnerships in Education Research (Research Partnerships) 
topic supports partnerships composed of research institutions and state or local 
education agencies that have identified an education issue or problem of high priority 

for the education agency that has important implications for improving student 
education outcomes. These partnerships are to carry out initial research and develop a 
plan for future research on that education issue. Through this joint research, the 
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education agency’s capacity for taking part in research and using results is expected to 
increase. The ultimate goal of the partnerships supported under this topic is to conduct 
and promote research during and after the grant that has direct implications for 
improving programs, processes, practices, assessments, or policies that will result 

in improved student education outcomes. 


The Research Partnerships topic provides funds to develop new partnerships and to 
support the expansion of existing partnerships into new areas of research. Partnerships 
are expected to complete initial research to help understand their education issues 

and develop a plan for future research. To this end, partnerships may analyze secondary 
data and/or collect primary data and analyze it. (p. 13) 


To develop an understanding of RPPs funded under the program, we surveyed and interviewed 
researchers and practitioners who were actively involved in the partnerships and analyzed their 
original grant applications. In this interim report, we present the results of the first phase of our 
study with regard to the following questions: 


Goals of the RPPs 

e What are the partnerships’ goals, and what progress have they made toward these goals? 

¢ Relative to partnerships’ other goals, how important is promoting research use? 
Conducting and Using Research in Partnerships 

¢ In what kinds of research projects are partnerships engaged? 

e What are practitioners’ patterns of research use? 

¢ Have researchers changed how they do research as a result of their partnerships? 
Activities of Partnerships 

e What types of activities have partnerships pursued to attain their goals? 
Communication Within Partnerships 

¢ How and how often do partners communicate with each other? 
Challenges to Working in Partnerships 

e What are common challenges to sustaining a partnership? 
Perceptions of the Partnerships 

¢ Do researchers and practitioners value long-term partnerships with each other? 
Planned Future Activities of Partnerships 


e What, if any, are the participants’ plans for continuing their partnerships beyond the end 
of the grant? 
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Guidance for IES 


¢ What feedback do RPP participants have for IES related to amount of funding, length of 
time of grant, level and type of monitoring, and opportunities for RPPs to meet to discuss 
work? 


¢ How might IES alter their RPP grant program to help future partnerships meet their 
goals? 


We first describe our instrument development process, sampling strategy, and key constructs. 
We then offer preliminary analysis related to the questions above. The findings of this study can 
not only inform the IES RPP program but also contribute to knowledge about the processes, 
successes, and challenges of RPPs in education. This work also provides information about 

the reported value of these collaborative efforts for researchers and practitioners interested in 
developing partnerships. 
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Conceptual Framework 


A set of key interlocking ideas guided the design of this study. We applied Weiss and Bucuvalas’s 
typology of research use® to address the question of how well the projects are meeting a core goal 
of the IES RPP program: building the capacity of educational organizations to use research. To 
inform the design of survey and interview questions related to goals and activities, we drew on 
the framework recently developed by Henrick and colleagues’ for characterizing the outcomes of 
RPPs as well as on a typology of research—practice partnerships. 


Types of Research Use 

In designing queries about leaders’ research use, we employed the categories Weiss and 
Bucuvalas’” first identified to capture the multifaceted nature of research use and that have been 
applied more recently in studies of educational leaders’ research use.'' This typology includes 
three types of use:'” 


¢ Instrumental use: Research is applied to guide or inform a specific decision. 


e Conceptual use: Research induces changes in the way a person views either a problem 
or the possible solution space for a problem. 


¢ Symbolic/political use: Research is used to validate a decision or legitimate a decision 
already made. 


Of these three, instrumental use is emphasized in contemporary education policies that are 
intended to promote research use. However, past research,’ as well as a recent study of research 
use among school and district leaders conducted by NCRPP,"* indicate that conceptual and 
symbolic uses are also common. Conceptual and symbolic uses can influence decision making 
indirectly by focusing attention on issues that were previously unknown to decision makers,!° by 
helping to identify opportunities for improving current programs and policies,'® or by building 
political support for local initiatives.'” 


Research-practice partnerships are hypothesized to promote use of all types because they provide 
opportunities for sustained interactions between researchers and practitioners around evidence." 
Sustained interaction is important to fostering research use because it involves the interactive 
processes of deliberation, persuasion, negotiation, and sensemaking.'? RPPs frequently involve 
structured activities to develop research and evaluation questions together and to make sense 

of results of studies of policies and programs.” Such processes may not only help practitioners 

to make sense of evidence, they may also be occasions for researchers to “give sense””! to the 
meaning of the evidence in light of how particular findings fit in with other research studies. 


Characterizing the Goals of IES Researcher-Practitioner Partnerships 

Promoting research use is not the only goal of RPPs, nor is it the only goal of the IES grant 
program. For example, many partnerships aim to design and test interventions together that can 
improve learning outcomes and be implemented at scale.” To characterize the goals of RPPs in 
this study, we draw on a recent effort by Henrick and colleagues funded by the William T. Grant 
Foundation.” 
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That effort, which used an iterative, participatory process of soliciting input and feedback from 
multiple RPPs across the country, yielded a typology of five categories of outcomes that the 

RPPs in the sample agreed were important to them to varying degrees. Though this framework 
encompasses some aims that might go beyond those explicitly promoted in the IES RPP 
program, we employed it here to capture the types of goals that the partnerships pursued. ‘The five 
categories from the framework are described below. 


Cultivate partnership relationships. RPPs aim to cultivate partnership relationships and build trust 
among researchers and practitioners. Productive working relationships are both the foundation 
for joint work and a consequence of working successfully together. Similarly, trust is a necessary 
ingredient in all partnerships,“ and it develops as people make commitments to one another and 
follow through on their commitments. 


Increase capacity of researchers and practitioners to conduct partnership work. Another goal that 
RPPs pursue is to develop the skills and dispositions of people to conduct work in partnerships. 
For researchers, this includes identifying problems of practice to study that could address gaps in 
foundational knowledge about learning or education. It also includes cultivating dispositions to 
listen to practitioners and seek out their expertise in diagnosing problems and designing solutions 
to them. For educators, it means cultivating an appetite for research evidence and developing 
skills necessary to participate in different aspects of the research process, from defining questions 
to providing feedback on instruments to making judgments about how best to apply findings in 
particular decision-making contexts. 


Conduct and use rigorous research. Just as researchers who engage in other forms of research 

and development do, researchers in RPPs aim to conduct studies that meet the highest standard 
of quality. When evaluating programs, for example, they seek to use the best designs available 
for estimating causal impact, including experiments.” What distinguishes RPPs from other 
forms of research is that they are focused sharply on problems of practice of their local partners. 
To that end, partnerships often produce descriptive studies that explore the relationships 

among malleable factors in educational environments,” as is one aim of the IES RPP program. 
Partnerships are also consumers of research. Sometimes, education leaders take up findings and 
use them to adjust policies and programs.” On other occasions, design teams composed of both 
researchers and practitioners make use of research analyses to refine their solutions to educational 
problems.” The capacity of educational organizations to conduct or use research may be further 
developed as part of this goal. 


Impact local improvement efforts. RPPs support educational organizations in making progress 
toward their local improvement goals. That is, the research they do is in service of larger 

aims for improving teaching and learning outcomes, not just to develop an understanding of 
problems. Sometimes, partnerships work together to identify and test strategies for addressing 
focal problems. Partnerships may also engage in continuous improvement research in which 
they develop, test, and refine particular strategies and use research evidence to refine or adjust 
those strategies.” Researchers in partnerships may develop systems of indicators to help their 
education partners track progress toward their program goals.*° 
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Inform the work of others. Although RPPs attend to local problems of practice, most also aim 
to inform the work of others outside those partnerships. Research conducted by RPPs can 
also contribute to new knowledge and theory that furthers our understanding of what it takes 
to support educational improvement across different educational settings.*! When given the 
opportunity, educators readily share knowledge, tools, and practices they have developed with 
other educators inside and outside their own organizations.” 


Describing the Activities of IES Researcher-Practitioner Partnerships 

It is typical for RPPs to engage in a broad range of activities, given the breadth and ambitions 
of their aims. Although the IES program is intended to support certain kinds of activities, such 
as descriptive research to refine an understanding of a focal problem, it is possible that some 
partnerships may be engaged in a broader range of activities. 


Different types of partnerships can be distinguished by the kinds of work researchers and 
educators do together and separately. A typology developed by Coburn and colleagues** depicts 
three broad types of partnerships. In research alliance models, RPPs engage in analyses of the 
implementation and outcomes of district policies and programs. Researchers share findings 

with educational decision makers and work with them to develop solutions (e.g., the University 
of Chicago Consortium on School Research). In design research partnerships, researchers and 
educational leaders co-develop and test strategies or tools for improving teaching and learning 
systemwide. They use an approach adapted from the learning sciences for conducting research 
on interventions in classroom, school, or district contexts. Still other RPPs organize as networked 
improvement communities (NICs) and engage in continuous improvement research to work on 
problems of practice. NICs are networks of people and organizations that can span multiple 
jurisdictions (e.g., districts, universities) and that are organized to achieve common improvement 
aims. 


As noted in the RFA, IES does not endorse a specific model of research partnerships. In the 
current study, we did not assume that each RPP would neatly fit into one of these three types. 
Instead, we hypothesized that a partnership may engage in a configuration of different activities 
based on the goals of their work together. For instance, a partnership could engage in activities 
that are more typical of research alliances, like integrating multiple datasets or performing 
independent analyses of district administrative data. The same group could also be involved in 
design work as they co-design and test strategies or tools for addressing identified needs. For this 
study, however, the typology helped to define the range of partnership activities we explored with 
respondents. 
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Study Design 


This study focused on describing the goals, activities, and perceived progress of participants in 
the IES RPP program as well as on their perceptions of the program. As a descriptive study, no 
inferences about the program’s impact can be made, nor was that the intent. This report shares 
descriptive findings from the first phase of a two-phase study. Below, we describe our population 
and sample, study methods, instruments, and the procedures for data collection and analysis. 


Population and Sample 

We studied the first three cohorts of RPPs funded by IES between 2013 and 2015. The population 
included a total of 28 RPPs and the active researchers and practitioners in those partnerships. 

By active members, we mean people with some responsibility for carrying out the work of 

the partnership. Researchers include principal investigators (PIs) and RPP members from 
universities and research organizations; this group does not include practitioners who were in 
research positions in practitioner organizations (e.g., a director of research in a school district). 
Practitioners include co-principal investigators (co-PIs) and all RPP members from practitioner 
organizations (e.g., school districts, departments of education, social service agencies). 


In order to identify the population of participants for the study, we began with the publicly 
available abstracts of the 28 RPPs. We aimed to include three to four active participants per 

RPP, with equal researcher and practitioner representation. Because only one-third of abstracts 
included information on three or more RPP members, and because it was possible that some 
turnover had occurred since the time of award, we pursued a snowball sampling approach to 
identify the currently active members of the RPPs.** We emailed the Pls of the 28 RPPs, asking 
them to share their original grant proposals with us, to confirm the active status of RPP members 
we had identified from the abstracts, and to name other key researchers and practitioners most 
involved in their partnerships. 


Original roster. Twenty-seven of the 28 PIs responded to our request. The PIs confirmed or named 
82 active researchers (including the 27 PIs) and 78 active practitioners (including 25 lead co-PIs). 
In the case of four RPPs where the PI named more than one co-PI, we asked them to identify 

a lead practitioner co-PI. For each of the responding 27 RPPs, our roster included the PI, the 

lead co-PI, an average of two other researchers, and an average of two other practitioners. We 
determined all rostered individuals’ professional positions via an online search and confirmed 
these positions during interviews when possible. Finally, we determined whether a rostered 
individual was in a senior-level position based on professional title and the structure of the 
research or practice organization. 


As shown in Table 1 below, the full roster consisted of 82 researchers, including 28 PIs and 54 
other researchers. The majority of PIs were professors (19; 68%), followed by researchers (6; 21%) 
and center directors (3; 11%). Many of the other researchers on the roster were professors (22; 
48%) or researchers with titles like research scientist (21; 41%), followed by doctoral students 

(7; 13%) and center directors (4; 7%). Over three-quarters of PIs (22; 79%) were in senior- 

level positions, such as associate or full professor, senior researcher, or director of a research 
organization. 
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Although more junior than the PIs overall, almost half of the other researchers were also in 
senior-level positions (26; 48%). 


Table 1. Rostered Researchers, by Position and Role in Partnership 


# (%) # (%) of other # (%) of all 

of Pls researchers researchers 
Professor a7 22 - 

(68%) (41%) (50%) 
Researcher 6 a a7 

(21%) (39%) (33%) 

: 3 4 7 
Center director (11%) (7%) (9%) 
0) 7 7 

Doctoral student (0%) (13%) (9%) 
Senior-level position a ae say 

(79%) (48%) (59%) 
Total 28 54 82 


As seen in Table 2 below, our roster included 78 active practitioners, which included 25 co-PIs 
and 53 other practitioners. The largest share of the co-PIs worked in positions related to P-20 
research, assessment, and accountability (12; 48%), followed by postsecondary policy and practice 
(3; 12%). Less than 10% of co-PIs represented each of the remaining areas of practice. 


The other rostered practitioners were somewhat more evenly divided, with most working in 
P-20 research, assessment, and accountability (14; 26%), followed by early childhood policy and 
practice (10; 19%); postsecondary policy and practice (9; 17%); and K-12 special education (6; 
11%). Less than 10% were in each of the remaining K-12 areas of practice. Four-fifths of the co- 
PIs (20; 80%) were in senior-level positions, such as directors of departments or organizations, 
cabinet-level leaders in school districts, or heads of educational agencies. Although more junior 
than the co-PIs overall, over half of the other practitioners were also in senior-level positions (30; 
57%). 
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Table 2. Rostered Practitioners, by Area of Practice and Role in Partnership 


# (%) # (%) 
# (%) of other of all 
of co-Pls practitioners practitioners 
P-20 research, assessment, 12 14 26 
and accountability (48%) (26%) (33%) 
Postsecondary policy and 3 9 12 
practice (12%) (17%) (15%) 
Early childhood policy and 2 10 12 
practice (8%) (19%) (15%) 
‘ ‘ 1 6 7 
K-12 special education (4%) (11%) (9%) 
2 5 7 
K-12 federal programs (8%) (4%) (5%) 
: 2 3 5 
K-12 educator evaluation (8%) (6%) (6%) 
K-12 curriculum and 1 4 5 
instruction (4%) (8%) (6%) 
K-12 deputy superintendents 2 2 4 
and chief officers (8%) (4%) (5%) 
Senior-level position 20 30 oo 
P (80%) (57%) (65%) 
Total 25 53 78 


Participants 

We gathered survey or interview data from 27 of the 28 IES-funded RPPs, for a partnership 
response rate of 96%. On average, there were four participants per RPP. Data for 25 of 27 
participating RPPs included at least one researcher and one practitioner. 


Overall response rates for rostered participants. Table 3 below details the response rates of the 
160 rostered individuals. A total of 106 participants completed the survey, the interview, or 
both, for an overall response rate of 66%. We received survey responses from 104 participants 
(response rate = 65%) and conducted interviews with 98 participants (response rate = 61%). 
Eight participants completed the survey but not the interview. Two participants completed the 
interview but not the survey; these two individuals started but did not fully complete the survey 
items, so they received an automatic invitation to schedule an interview. 


Overall, response rates were higher for researchers than for practitioners. Of the 82 rostered 
researchers, 61 (74%) completed the survey, the interview, or both, while 45 (58%) of the 78 
rostered practitioners did so. Among researchers and practitioners, participation was high for 
PIs (26; 93%) and for co-PIs (21; 84%), and lower for other researchers (35; 65%) and other 
practitioners (24; 45%). 
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Table 3. Response Rates for Pls, Other Researchers, Co-Pls, and Other Practitioners 


# of participants 
(response rate) 


Survey, interview, 


Rostered Survey Interview or both 
104 98 106 
al oe (65%) (61%) (66%) 
Researchers 82 e 28 ie 
ane (74%) (68%) (74%) 
26 24 26 
PI 28 
7 (93%) (86%) (93%) 
35 32 35 
Oth h 54 
er researchers (65%) (59%) (65%) 
Practiti r 78 id is i 
Reearene (55%) (54%) (58%) 
20 19 21 
corre 43 (80%) (76%) (84%) 
7 23 23 24 
Other practitioners 53 (43%) (43%) (45%) 


Response rates by position and senior level. The 61 researchers who completed the survey, 
interview, or both served in a range of positions. Thirty-four (56%) were professors, and 19 
(31%) were researchers with titles like research scientist. Fewer of the participating researchers 
(4 each) were center directors (7%) or doctoral students (7%). A chi-square test indicated that 
the proportions of respondents in each position relative to all responding researchers was not 
significantly different from proportions of each position in the original roster (X?[3, n = 61] = 
1.016, p = 0.798). That is, the proportions of participating researchers by position mirrored the 
proportions of all researchers in the original roster. (See Appendix A for detailed proportions of 
respondents and non-respondents by position and partnership role.) 


The 45 responding practitioners represented the following areas of practice: Nineteen (42%) were 
in P-20 research, assessment and accountability; eight (18%) worked in postsecondary policy and 
practice; equal numbers (5 practitioners each) worked in either K-12 special education (11%), 
K-12 federal programs (11%), or early childhood policy and practice (11%); two (4%) worked in 
K-12 educator evaluation; and one (2%) worked as a K-12 deputy superintendent or chief officer; 
none held a position in K-12 curriculum and instruction. 


Although some areas of practice had lower response rates than others (e.g., K-12 deputy 
superintendents and chief officers; K-12 curriculum and instruction), it is important to keep in 
mind the low numbers of individuals rostered for each position. A chi-square test indicated that 
the proportions of respondents in each position relative to all responding practitioners were not 
significantly different from the proportions of each position in the original roster (X?[7, n = 45] = 
6.114, p = 0.527). 
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As they did among researchers, then, the proportions of participating practitioners by area of 
practice mirrored the proportions of all practitioners in the roster. (See Appendix A for detailed 
proportions of respondents and non-respondents by position and partnership role.) 


Finally, of the 48 senior-level researchers in our original roster, 36 (75%) completed a survey, 
interview, or both. Of the 50 senior-level practitioners in the original roster, 30 (60%) completed 
the survey, interview, or both. Chi-square tests indicated that the proportions of participating 
researchers and practitioners in senior-level positions mirrored those in the full roster (X?[2, n 

= 45] = 0.377, p = 0.828) and (X’[6, n = 45] = 0.417, p = 0.999), respectively. The response rates 
of those in senior-level positions generally mirrored those of all participants in those positions. 
(See Appendix B for more details on response rates by position, area of practice, and senior-level 
designation.) 


In sum, while some response rates were higher than others for different groups, the fact that the 
proportions of positions mirrored the overall roster along these different lines (i.e., positions 

or areas of practice, senior-level roles) increases our confidence that the sample is a fair 
representation of the rostered population. 


Survey participant demographics. Table 4 shows the demographic information collected for the 
104 survey respondents. Both researchers and practitioners were experienced in their positions 
and held advanced degrees. On average, PIs had served almost 15 years in their positions, 
compared to eight years served by other researchers. Co-PIs and other practitioners had served 
over eight years on average in their positions. Almost all PIs (96%) and other researchers (89%) 
held doctoral degrees. Notably, half of co-PIs (50%) and 39% of other practitioners also held 
doctoral degrees. Across all groups, participants largely identified as female (66%) and White or 
European American (83%). 
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Table 4. Demographics of Survey Respondents 


Researchers Practitioners 
Other Other 
All Pls researchers | Co-Pls practitioners 
(n=104) (n=26) (n= 35) (n = 20) (n = 23) 

Average years in current role 9.9 14.8 8.0 8.5 8.3 
Highest degree held 

Doctorate 72% 96% 89% 50% 39% 

Master’s 25% 4% 9% 45% 57% 

Bachelor’s 3% 0% 3% 5% 4% 
Gender identity 

Female 66% 58% 66% 55% 87% 

Male 34% 42% 34% 45% 13% 
Racial or ethnic identity 

White or European American 83% 81% 89% 85% 74% 

Latino(a)/Hispanic 6% 4% 0% 15% 9% 

oe ee ee 

Asian American or Pacific Islander 4% 4% 9% 0% 0% 

Multiracial 2% 4% 3% 0% 13% 

Arab American or Indian American 1% 4% 0% 0% 0% 

American Indian or Alaskan Native 0% 0% 0% 0% 0% 


Note. Gender identity was asked as an open-ended question; racial or ethnic identity options correspond to those on 
the U.S. Census; because demographic data were gathered on the survey, the table does not include information for two 
participants who completed the interview but not the survey. 


Sources of Data 
There were three sources of data for the study: grant applications, surveys, and interviews. 


Grant applications. Each partnership submitted a grant application in order to be considered for 
IES funding. We used these applications as a source of data regarding each partnership’s history, 
as well as its research questions, methods, and data sources. In the application, the team described 
the strengths of each partner organization, the overall strength of the partnership, and their plans 
for developing the relationship over time. Each applicant outlined the partnership's research 

aims, the education issue to be addressed, the importance of the issue for all parties involved, and 
details about their research plans. Applicants were asked to explain their plans for activities aimed 
at supporting/increasing the research capacity of the practitioner organization(s) as well as plans 
for the partnership’s future work together. 
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Survey. We developed two survey instruments, one for researchers and one for practitioners. 
Surveys included five previously tested scales of items from NCRPP’s national survey of 
educational leaders’ research use and attitudes, including instrumental, conceptual, and 
symbolic uses of research, as well as new items specific to this study. New items were tested and 
revised through a cognitive piloting process with two practitioners and two researchers. We also 
solicited expert feedback from advisors. We then made revisions to items on the basis of team 
discussions that took into account the pilot results and expert feedback. The constructs listed 
below are those included in this report; other constructs on the survey (e.g., organizational 
culture) will be included in future reports. 


Definition of research in the survey. Each page of the survey that included items related to research 
provided respondents with a definition of research as “an activity in which people employ 
systematic, empirical methods to answer a specific question.” The first page of the survey provided 
the following elaboration: 


Research bases its conclusions in investigations involving statistical data, interviews, 
observations, and case studies, or a combination of these. Research can appear in 
books, academic journal articles, practitioner-oriented journals, and analyses of 
program implementation developed by researchers external to the district. It can also 
appear in policy and evaluation reports or presentations developed by researchers 
within a district. 


For this study, we differentiate between research, which involves systematic inquiry 

to answer a specific question, and the practice of looking at data from the district, 
school, or classroom, which is more open-ended and seldom addresses specific 
research questions. For instance, looking at state standardized test results to identify 
students who need extra support in the classroom would not be research. However, 
asking the question, “What is the relationship between fourth-grade state standardized 
test results and high school graduation?” would be research. 


NCRPP scale: Instrumental research use. Instrumental use occurs when research is applied to 
guide or inform a specific decision. To elicit practitioners’ instrumental use of research, we 
provided a list of six decisions common to educational organizations, including curriculum 
adoption, scaling up a pilot program, designing professional development, and other activities. 
We first asked participants if they had been involved in each type of decision. Because this 

study surveyed practitioners in a range of educational organizations (not only school districts), 
for the instrumental use scale we omitted one item that asked about involvement in adopting 
curriculum materials. We also condensed two items on designing professional development for 
administrators and teachers to one item on designing professional development. Respondents 
who indicated they were involved in an activity were then asked how often they had used research 
as part of that activity. This scale was adopted from the NCRPP research use survey where it 
demonstrated excellent reliability (a = .93). We also asked practitioners whether the partnership 
was consulted or involved in that activity. Item response choices were: Never (1), Sometimes (2), 
Frequently (3), All of the time (4). This item bank only appeared on the practitioner version of the 
survey. 
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NCRPP scale: Conceptual research use. Conceptual use refers to research that is applied to induce 
changes in the way a person views a problem or space of possible solutions. In our survey, we 
sought to elicit the extent to which research informs practitioners’ ways of looking at problems 

or their approaches to solving district problems. The six-item scale included questions such as, 
“How often have you encountered research that changed the way you look at problems facing 
your school/district?” and “How often have you encountered research that suggested alternative 
solutions to a district problem?” As with the instrumental use scale, item response options were: 
Never (1), Sometimes (2), Frequently (3), All of the time (4). This scale was adopted from the 
NCRPP research use survey, where it demonstrated good reliability (a = .88). This item bank only 
appeared on the practitioner version of the survey. 


NCRPP scale: Symbolic research use. Symbolic use, sometimes referred to as political use of 
research, occurs when research is leveraged to influence a decision or to legitimate a decision 
that has already been made. The four-item scale asked respondents to report their engagement 
in activities such as using research to mobilize support for important issues or selectively using 
research to support decisions. Item response choices were: Never (1), Sometimes (2), Frequently 
(3), All of the time (4). This scale was adopted from the NCRPP research use survey, where it 
demonstrated good reliability (a = .81). This item bank only appeared on the practitioner version 
of the survey. 


Goals. Partnerships may have multiple aims that drive their joint efforts. To elicit respondents’ 
reports of partnership goals, we first provided a list of 14 potential goals, based on prior 
research and preliminary analysis of the grant applications. Goals included plans to identify a 
specific strategy for improvement, improve students’ academic outcomes, or build a foundation 
or infrastructure for future work together. We asked whether this was a current goal of the 
partnership, was a goal at the start of the partnership, or was not a goal. If a respondent 
indicated this was or had been a goal, we subsequently asked what progress the partnership 
had made towards that goal. Item response choices were: No progress (1), A little progress (2), 
Some progress (3), Accomplished (4), or Exceeded (5). This set of items was included on both 
researcher and practitioner surveys. 


Partnership activities. Participants of RPPs can interact with each other for a range of purposes. 
We developed a list of 18 potential activities of partnerships, drawing on our conceptual 
framework and a preliminary analysis of project abstracts. Example activities included: develop 
a set of conjectures or hypotheses about features of programs that are needed to address a 
problem; build a data archive or database that combines multiple data sets; or create a networked 
improvement community of people or other organizations interested in testing solutions to 

a problem. Item response choices were: Not part of this grant (1); Has done as part of grant 

(2); Plans to do as part of grant (3); Plans to do after grant concludes (4). This set of items was 
included on both researcher and practitioner surveys. 
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Communication strategies. We asked participants to report how frequently they communicated 
with their partners. Modes of communication included email, phone, or text; working on shared 
documents or an online platform; in-person meetings; teleconference or video meetings; and 
other. Item response choices were: Never, Yearly, Bi-monthly, Monthly, Weekly, or Daily. This item 
bank was included on both researcher and practitioner surveys. 


Attitude items related to the value of partnership. We included six items with respect to the 
value of the partnership and plans for future work. These items included statements like, “T 
would participate in a researcher-practitioner partnership in the future” or negatively phrased 
statements such as, “I would not recommend to a colleague that they join or form a researcher- 
practitioner partnership.” We asked respondents to indicate to what extent they agreed or 
disagreed with each statement, using these item responses: Strongly disagree (1); Disagree (2); 
Agree (3); Strongly agree (4). This set of items appeared on both researcher and practitioner 
surveys. 


Challenges. These are the set of barriers that partnerships can face in their work together. We 
asked respondents to indicate the degree to which they experienced different barriers identified 

in the literature on research—-practice partnerships. There were eight items in all. Topics included 
difficulties in communicating across boundaries of research and practice, asynchrony in timing 

of joint work, difficulty navigating unfamiliar roles, and turnover and change in leadership in a 
partner organization. We asked respondents to assess the extent to which they agreed or disagreed 
with each statement, using these response choices: Strongly disagree (1); Disagree (2); Agree (3); 
Strongly agree (4). This bank was included on both researcher and practitioner surveys. 


Guidance to IES. The RPP program has current design features that could be revised in the future. 
This set of four items asked about RPP members’ perceptions of the level of funding, the length 
of time of the grant, and opportunities to interact with other partnerships. We asked respondents 
to assess the extent to which they agreed or disagreed with each statement, using these response 
choices: Strongly disagree (1); Disagree (2); Agree (3); Strongly agree (4). 


Demographic information. Finally, we asked participants to report their own demographic 
characteristics related to years of experience, educational degrees earned, gender, and race or 
ethnicity. 


Interview protocols for researchers and practitioners. We developed two semi-structured interview 
protocols, one for researchers and one for practitioners, both of which we piloted before 
beginning interviews. Each protocol included 22 questions related to the individual's role in the 
partnership, the partnership's goals and progress made toward those goals, how the partnership 
compared to previous experiences of interacting with researchers or practitioners, challenges to 
sustaining the partnership, and guidance to IES about the program. 
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Data Collection Procedures 

We administered the survey via Qualtrics, an online survey administration platform, with an 
eight-week completion window for each respondent. In June 2016, IES leadership sent an email 
to all PIs, introducing the study. We then sent an email to each contact on the roster with an 
invitation to participate in the study and a link to the appropriate survey on Qualtrics. We sent 
three reminder emails over three weeks and followed up with one to two phone calls to survey 
non-respondents. In some cases, we delayed the reminder if we learned that the respondent was 
unavailable for periods of time over the summer. 


Surveys took 15 to 20 minutes to complete. Once they started the survey, participants 
automatically received an email with a link to schedule a phone interview. Interview scheduling 
involved the same pattern of three emails at weekly intervals and one to two phone call reminders. 
All but eight of the 104 survey respondents scheduled interviews with one of the members of the 
research team, which lasted 45 minutes on average. Therefore, unless a person directly declined 

to participate, he or she could receive up to eight emails and four phone calls over the 10-week 
period to invite their participation in the survey and interview. At that point, we determined that 
no response was a signal for non-participation. 


We aimed to have both researcher and practitioner perspectives represented in as many RPPs as 
possible. For 25 of the 27 participating RPPs, we secured a minimum desired response of at least 
one researcher and one practitioner for each RPP, for both the survey and interviews. The other 

two RPPs had only researcher participation. 


Issues of Confidentiality and Anonymity 

In this report, we do not identify any individual partnerships, and we have changed all names 
to pseudonyms. We have avoided using any identifiable language related to the individuals or 
the partnerships. Our reporting here focuses on broad themes and patterns that emerged across 
partnerships, not on reports of individual partnerships. 


Approach to Analysis 

To answer our research questions, we first looked at each source of data separately, producing 
descriptive statistics from our analysis of proposals and surveys and coding summaries of 
interview data. We considered carefully what each data source was well-suited to help us 
describe concerning the partnerships. For instance, the proposals have the most information 
about research design. We also considered where data sources might complement one another 
and where we could triangulate our initial conclusions. For example, proposals and interviews 
together were a good source of information regarding partnership history. We discussed analyses 
in weekly team meetings and developed a preliminary set of claims that we iteratively refined with 
the help of evidence matrices organized around those claims.** Below, we describe the initial set 
of analyses of proposals, surveys, and interview transcripts. 


Analysis of proposals. Three research team members designed a systematic grant review process, 
which began with a review of six applications to identify common components. Team members 
then agreed upon low-inference codes that corresponded to major components of the IES grant 
applications, and developed a coding guide with a detailed definition and at least one example for 
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each code. The key codes included the following: prior experiences of working together, goals of 
the partnership, topic of the focal problem of practice, type of research question, sources of data, 
research methods employed, and planned activities. 


Using Dedoose, a data management and analysis program for mixed-methods research, two team 
members separately coded two applications and then compared and reconciled their use of codes. 
Following this calibration, the 25 remaining applications were divided among researchers, with 
each researcher independently coding 12-13 additional applications. The team met regularly to 
discuss any issues that arose during grant application coding in order to ensure consistency. 


Survey analysis. After survey data cleaning, one research team member calculated descriptive 
statistics for reported survey constructs. In order to investigate agreement between groups within 
partnerships, we compared responses among PIs, co-PIs, other researchers, other practitioners, 
all researchers, and all practitioners. To investigate similarities and differences across RPPs, we 
determined whether at least 50% of each RPP’s participants gave consistent responses for items or 
banks of items. 


Interview analysis. Interviews lasted 45 minutes on average and were recorded and transcribed. 
Research team members reviewed interview transcripts for accuracy, replaced individuals’ names 
with pseudonyms, and uploaded cleaned transcripts to Dedoose. Team members tagged each 
transcript with descriptors for the data collection time period (summer 2016); participant type 
(practitioner, researcher, PI); and RPP (by number) so as to be able to track individuals and 
partnerships. 


We proceeded in an iterative fashion to analyze the qualitative interview data.” First, we engaged 
in low-inference coding. The goal of this initial stage of coding was to index interviews by key 
constructs like “Guidance to IES” or “Goals.” Lead researchers created a coding guide with 

major constructs, definitions, relevant interview questions, examples, and non-examples for 

each code. This coding guide was revised as the team coded transcripts, first together and then 
independently once they had reached a shared understanding of codes. A team of five team 
members engaged in this initial coding. Lead team members periodically checked coding to 
ensure integrity to the coding guide, and the team met regularly to discuss any issues that arose. 


Next, for each specific line of analysis, lead team members conducted a more refined analysis 

of coded data to identify themes. Starting with the relevant coded excerpts, two team members 
began with an a priori, deductive coding list based on the current literature base. They added 
codes inductively based on themes that emerged from the data. For instance, for the “Guidance 
to IES” category, a priori codes included funding, grant length, monitoring, and opportunities 

to engage with other partnerships. Within the funding code emerged subcodes about “enough 
funding,’ “requests more funding,” or “need for continuation grants.” A third coder reviewed all 
coding for consistency. We analyzed patterns from interview coding within matrices, considering 


patterns by individual, role group, or partnership. 


Because of time constraints, interview analysis for this interim report focused on patterns from 
the most active representatives of the partnerships. Here, we draw on 54 interviews from 28 
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researchers and 26 practitioners across the 27 RPPs in our sample. For 23 of the 27 RPPs, this 
analysis included the PI and the most active practitioner partner (most often the lead co- 

PI). For two RPPs where we determined from the interviews that the PI played an advisory 

role, we included a second researcher who was centrally involved in project management and 
leadership. There was one partnership where we could only include the PI interview and a second 
partnership where we could only include a practitioner interview. 


Study Limitations 

One limitation to the study is that we rely on retrospective self-reports from surveys and 
interviews. Self-reports of socially desirable behaviors, like research use, are always subject 

to bias. Further, the fact that they are retrospective makes them subject to accounts that gloss 
over challenges, dilemmas, and uncertainties in decision making. Research use and the role 
for partnerships are interactive phenomena that are best understood using multiple methods, 
including observations of partners’ work together and the role of research therein. 


Second, there are some caveats to our sample. We identified three to four individuals who were 
key participants in each partnership through a focused snowball sampling technique with the 
Pls and the publicly available abstracts. However, the perspectives of members who may be more 
peripheral may be different, and they are not currently included here. Further, likely because 

of the summer data collection window, we had lower response rates for practitioners. Finally, 
these findings on IES’s 27 partnerships may not be generalizable to all other research—practice 
partnerships, although they can contribute to theory building related to this field of work. 
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Description of the IES Researcher-Practitioner Partnerships 


Based on our analysis of grant applications, the 27 RPPs varied in terms of their start dates, prior 
relationships, organizational configurations, central educational issues, and focus of their joint 
research projects. 


The RPPs that participated in our study had been awarded two-year grants in the first three years 
of the IES RPP program, with six RPPs beginning in 2013, 13 in 2014, and eight in 2015. Due to 
no-cost extensions granted to some RPPs, at the time of this report, 11 were due to end in 2016, 
and 16 were due to end in 2017. 


Even though the program does not specify that partnerships should have prior experience 
working together, all 27 had done so in some capacity prior to receiving funding from 
IES, whether as part of individual projects or through informal or traditional data-sharing 
arrangements (see Table 5 below). 


Table 5. Examples of Prior and Formal Relationships in RPP Grant Applications 


All partnerships had previous relationships through... Examples from grant applications 


“In summary, several members of the partnership have 
worked together on past projects related to [policy area]. 
However, most of this work has been done on a volunteer 
basis in addition to the often considerable job duties and 
Prior work on individual projects responsibilities of the partners. The funding provided by 
the proposed grant would allow the partners more time to 
focus on the dual goals of building a formalized, sustainable 
partnership and conducting preliminary research to guide 
future research and intervention efforts.’ 


“The researcher practitioner project proposed here 
follows several years of a more traditional relationship 
between these Pls, mainly one where the [university] 
researchers had access to [educational agency] data for 
research purposes, but had minimal contact with those at 
the [educational agency] providing the data.” 


Informal or traditional data sharing arrangements 


Ten partnerships had formal arrangements marked by: 


“Over the last two years this partnership has demon- 
strated its ability to align the interests of practitioners 
and researchers to work together around a shared set of 
interests to build an analytic database and employ that 
database for a preliminary analysis of [policy area]....Over 
the last two years we have signed a data memorandum 

of agreement that led to data sharing and the continuing 
development of an analytic database.” 


A formal data-sharing agreement 


“({Educational agency] and [research agency] would expand 
upon a successful and mutualistic research partnership. 
This unprecedented investment comes on the heels of a 
long-term collaboration between [educational agency] and 
[research agency.] The Partnership stems from a 10-year 
relationship between [PI] and [educational agency], which 
was formalized through a 3-year IES Research-Practi- 
tioner Partnership Grant initiated in 2013.” 


A long-term, mutualistic arrangement between agencies 
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Ten RPPs described arrangements with formal data-sharing agreements or already formalized 
RPPs. In the second phase of this study, we plan to ask all participants when their joint work 
started, when they felt they became a “partnership, and what the indicators of that are to them. 


The partnerships funded by IES included a combination of research organizations and 
educational organizations. Research organizations included universities or other research agencies 
like the RAND Corporation or SRI International, while practitioner organizations included 
school districts, departments of education, higher education organizations, or other educational 
or social services agencies. Three-quarters (21) of the 27 RPPs included two organizations, while 
the remaining RPPs involved more than two. Close to two-thirds (18) of the RPPs included at 
least one university partner, while the same number had at least one school district partner. 


Each RPP in our study intended to address a central educational issue (see Figure 1 below). 

The most frequently named issues—cited by 12 of the 27 RPPs—related to K-12 teaching 

and learning, with five focusing on improving curriculum and instruction, four on academic 
achievement, and three on social-emotional or non-cognitive skills and outcomes (e.g., social- 
emotional well-being, mindset, disproportionality in discipline). In addition, three RPPs 
addressed issues of K-12 teacher quality or evaluation, and two centered on K-12 school 
improvement. Four RPPs identified early childhood education as their main issue, five pursued 
postsecondary access and success, and one focused on coordination across state service providers 
and education agencies. 


Curriculum & Instruction 

Post Secondary Access and Success 
Early Childhood Education 

Academic Achievement 

Teacher Quality/ Evaluation 

Social Emotional/ Non-Cognitive Skills 
K-12 School/ System Improvement 


Coordination Across Agencies 


Figure 1. Central educational issues of partnerships, from awarded RPP grant proposals (n=27). 
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Goals of the Partnerships 


Consistent with the purpose of the RPP program, nearly every application (24 of 27) stated 

goals related to developing practitioners’ capabilities to conduct or use research or to exploring 
problems through rigorous research activities. These were not the only goals that partnerships 
pursued, however. In the surveys and interviews, we found strong evidence that the RPPs also had 
explicit goals to cultivate partnership relationships, to impact local improvement efforts, and to 
inform the work of others. 


Key Findings: 


e In the original grant applications, the most commonly reported goals were those emphasized 
within the RFA for the program: conducting and using research and impacting local improvement 
efforts. 


e RPPs also had explicit goals to cultivate partnership relationships, to increase capacity of 
researchers and practitioners to engage in partnership work, and to inform the work of others. 


e Insurveys, participants reported making the most progress toward the goals of developing the 
foundation and capacity for partnership work, conducting and using research, and identifying and 
evaluating improvement strategies. 


e Insurveys, participants reported making somewhat less progress toward goals related to 
improving organizational policies and processes and improving student outcomes. 


Conducting and Using Research 

In the interviews, participants from all 27 RPPs described goals related to conducting and using 
research to address their focal problems. There was also agreement between researchers and 
practitioners around this goal within partnerships, with 18 RPPs having at least one researcher 
and one practitioner naming it in interviews. Survey results corroborated this finding, with 

at least half of the participants in every RPP reporting goals related to conducting and using 
research. (See Figure 2 below; for goals broken out by respondent group, see Appendix C.) 


Partnerships pursued goals of conducting and using research by creating multi-sector datasets, 
conducting secondary analyses of student-, teacher-, and school-level data, conducting interviews 
and focus groups, creating case study reports, and more. As one practitioner described her 
partnership's work: 


Essentially, we looked at over a decade of administrative data on students. We were able 
to determine their probabilities of meeting certain academic outcomes. Then, based 

on students who had been successful in the past of achieving those outcomes, we could 
look at current students based on their characteristics, and come up with the probability 
that they were going to meet those particular outcomes. We found one of the key lowest 
outcomes we had was for kids who were successfully achieving in math. That gave 

us the basis for designing this math intervention program this continuous 

improvement program, as we move forward....We had hard data to really validate 

that this was one of the areas we should focus on. 
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In some cases, as described above, both researchers and practitioners worked together on data 
collection or analysis. In other instances, researchers took the lead on data analysis (e.g., when 
it involved advanced statistical methods), and practitioners were more involved in subsequent 
sense-making and determining next steps. 


Participants sometimes described this aim of conducting and using research in terms of 
developing practitioners’ abilities to engage with research. Rather than sharing highly 
technical statistical methods, this involved a goal for the educational organization to become 
more “evidence-based” in their work more generally. These aims sometimes also focused on 
practitioners becoming more “discriminate users[s] of research,” as one practitioner noted. 
Another practitioner from a different partnership similarly suggested: 


We don't necessarily have to build statisticians out of everybody. It’s useful to be able to 
use research and to understand research, but it’s not realistic or optimal to have 
everybody be able to actually create it. In terms of helping policymakers and 
practitioners use it, it’s the guidance on, “Here's how to vet good research. Here’s how 
you know to trust it.’ 


Finally, with 42% in positions related to research, assessment, and accountability and 44% with 
doctoral degrees, practitioners often already had substantial knowledge of research methods. 
Rather than building individual capacity to use research, many participants explained that the 
goal of their work was to build the capacity of educational organizations to engage with research 
and to better integrate the work of researchers with other departments. One researcher, for 
instance, explained that the goal of the partnership was to help reestablish the role of research 
within the organization after years of budget cuts: 


[The state] went through an extended period of either stagnating K-12 funding or falling 
K-12 funding. In this district, the size of the staff that was involved in research 

and policy evaluation dwindled sharply over time. There are still many people with quite 
sophisticated knowledge of social science research methods, but they just don’t have the 
time and the wherewithal to do policy research. We're helping them answer questions 
they want answered and probably would've been in the position to answer 15 to 20 years 
ago, but due to cutbacks, just don't really have the time to do it. A lot of their time is spent 
working on mandatory evaluation reports for state, federal, and other funders. I don't 
think it’s really changed them, but I think it’s allowed them to sustain one of the 

things that got them into education policy in the first place, which was doing research. 


Impacting Local Improvement Efforts 

In addition to conducting and using research, all 27 RPPs had a member who, in interviews, 
identified impacting local improvement efforts as one of the most important goals of the 
partnership. Again, there were high levels of agreement within partnerships, as 21 partnerships 
had at least one researcher and practitioner who discussed their local improvement efforts. On 
the survey, all 27 partnerships had at least half of participants agreeing that “developing a deep 
understanding of the focal problem” was a goal of their work. 
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Often, participants began with data analysis to develop a better understanding of the problem 
of practice at hand. Within the areas of focus named above, partnerships focal problems varied 
widely. Some focused on improving outcomes for specific groups of students, such as emerging 
bilingual learners (EBLs), students living in foster care, or students living in poverty. Others 
focused on improving policies and programs related to teacher effectiveness, postsecondary 
access and success, or early childhood education and kindergarten readiness. 


Further, partnerships sought to impact local improvement efforts beyond developing a deeper 
understanding of their focal problems. That is, beyond just having a better sense of the issue at 
hand, 77% (20) of PIs who responded to the survey said that a goal of their partnership was to 
improve students’ academic outcomes, and 69% (18) said they hoped their partnership would 
improve organizational processes in ways that would impact student outcomes. 


Increasing Capacity to Conduct Partnership Work 

In our interviews, at least one member of each of the 27 RPPs described increasing the capacity 
of researchers and practitioners to conduct partnership work as an important goal. There was less 
consistency of reporting across role groups, though; in only eight RPPs did at least one researcher 
and one practitioner both name the aim of increasing capacity to engage in partnership work. 


Nearly all partnerships were focused on increasing practitioners’ opportunities to use research, as 
noted above. However, interview data from 10 partnerships also revealed a focus on developing 
researchers’ skills to work in partnership with practitioners in order to develop more relevant and 
impactful research. As one researcher put it: 


It really is consistent with the mission of [the research organization] in training 
researchers to do rigorous research that makes a difference. It’s very much a part 
of what I consider my job as a professor and my own personal goals about wanting 
to do policy research that makes a difference. 


In this way, researchers and practitioners talked about learning new ways of engaging in 
partnership work together that would involve both developing new practices. As another 
researcher explained: 


For me, it is a learning process. I’m enjoying learning more about how to collaborate with 
[practitioners]. Some of those lessons are probably generalized, but some of them may be 
specific to the specific people in [this city]. It isn’t [that] I'm going up there and presenting 
results and, “Oh, look how smart I am.’ It’s going up there and sharing some results, but 
also having them come back and say, “But that doesn't make sense to us because this is 
the way we understand that.” We say, “Oh, wow, okay, let’s go rethink that”. . It’s 

this openness of sharing ideas and trying to think about how best to address those ideas. 


The ways in which RPPs thought about increasing capacity to conduct partnership work, then, 
took into account opportunities for practitioners to engage with research and for researchers to 
become collaborators around important problems of practice. 
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Cultivating Partnership Relationships 

In their interviews, at least one member from over three-quarters of the RPPs (20 of 27) identified 
“cultivating partnership relationships” as a primary goal of the partnership. At least half of all 
survey respondents in each RPP—including all of the Pls—reported that building a foundation 
for work together was a goal of their partnership. 


To do so, partnerships attended to foundational needs such as identifying individuals to play key 
roles in the partnerships, establishing data sharing agreements, dedicating staff to RPP efforts, 
and taking time to build trust and relationships. In addition, newer RPPs devoted time to learning 
about the issues that concerned partnership practitioners and the resources that might be brought 
to bear on the problems they faced. As one researcher explained: 


The goal for this initial partnership was...to really develop a better understanding of how 
we would be of value to them and develop the relationship. To figure out, how do they 
want to consume results? What types of methods do we use that are of benefit to them? 
What types of data do they have, what types of issues are they concerned about that we 
might be able to look into? What types of funding resources might we be able to jointly 
pursue? It’s about getting a better understanding of who our partner was and how we 
could be the best possible partners. 


Just as this researcher described the familiarization needed for an “initial partnership,” overall, 
RPPs in which partners had less previous experience working together might be expected to put 
more effort into cultivating relationships than those with more experience working together. This 
is a conjecture we plan to explore in future analyses. 


Informing the Work of Others 

Partnerships were not just concerned with solving local problems. In 25 RPPs, at least half of 
the participants agreed that developing findings that apply to other organizations was a goal of 
their partnership. In the interviews, informing the work of others—both inside and outside of 
academia—was identified as a goal by at least one member of 19 RPP teams. This goal tended 
to be supported especially by researchers—in 19 RPPs a researcher raised it as a goal, while 
practitioners raised it in only seven of these partnerships. 


Interview analysis suggests that partnerships sought to inform the work of others both through 
traditional publication venues and through efforts to reach practitioner audiences outside of the 
partnerships. One researcher noted that the partnership had succeeded in having some papers 
published in research journals, but that they were excited that their work was reaching a different 
audience because Scholastic Magazine reported the findings: 


...and that led to a presentation at [a conference] that was full of school-system, 
human-resource type people. I feel like it’s reaching a different kind of audience than 
the typical academic audience. 


In addition to conference presentations, participants reported developing workshops, videos, 
assessment and survey instruments, and research briefs to share with broad audiences. 
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Figure 2. Goals reported by at least half of members of each RPP in survey, n=27 partnerships. 


Progress Toward Goals 

We asked partnership members about their progress toward the goals they had identified on the 
survey. Overall, the highest levels of progress reported by PIs related to building a foundation of 
work together, developing an understanding of the focal problem, addressing the focal problem 
through research activities, and developing evidence for publication in a peer-reviewed journal. 
Somewhat less progress was reported for goals related to improving organizational policies and 
processes and improving student outcomes. 


Figure 3 shows the reports of average progress for each goal identified on the survey, in which 
a “4” represents a benchmark of having accomplished the goal. We show overall averages for 
all cohorts because, with only two exceptions related to outcomes for students, more recent 
partnerships did not differ from the first cohort with respect to reported progress. 


Technical Report No. 2 30 
National Center for Research in Policy and Practice 


od 
o 


Build foundation for future work together 


Develop understandings of how researchers 
and practitioners learn from each other 


wo 
nn 


oo 
nn 


Develop researchers capacity to work in partnerships 


oe 
ns 


Address focal problem through research activities 


Develop organization’s capacity to use research 


@ 
BK 


Develop evidence for publication 
in a peer-reviewed journal 


oo 
Hn 


Develop deep understanding of focal problem 


| 
Ne) 
wo 
BK 


Improve students’ academic outcomes 


Identify or evaluate an improvement strategy 


N 
N 


Improve organizational processes or policies 


N 
i) 


Improve students’ social-emotional/ 
non-cognitive outcomes 


N 
an 


Develop findings that apply to other organizations 


wo 
oo) 


0.0 0.5 1.0 aS: 2.0 2:5 3.0 3.5 4.0 
@ Cultivate partnership relationships ® Increase capacity to conduct partnership work 
™ Conduct and use rigorous research = Impact local improvement efforts 


@ Inform the work of others 


Figure 3. Progress towards partnership goals, in survey, n=104 survey respondents. 
1=No progress; 2=A little progress; 3=Some progress; 4=Accomplished; 5=Exceeded 


The progress reported in the survey is consistent with what we know about the development 

of partnerships.** Building trusting relationships enables partnerships to take on the work 

of deepening understanding of problems and engaging in joint work to impact policies and 
programs. Impacts on teaching and learning are important but they are lagging indicators that 
depend on new policies and programs. Though Figure 3 offers only a snapshot in time, the cross- 
sectional data are consistent with this developmental progression, in that we observe less progress 
toward lagging indicators and more progress toward building the foundations for partnership 
work. 


Of note is that both surveys and interviews revealed that many practitioners began with a 
relatively high capacity to conduct research, a fact that might explain why progress on the goal of 
building capacity of practitioners is not at the top of Figure 3. Thus, as participants described to 
us, the partnership built on practitioners’ existing understanding of research methods in addition 
to bringing their deep local knowledge to the work. 
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As one researcher put it, the partnership gave practitioners an opportunity to learn more 
advanced statistical approaches that could help them explore their measures more deeply. 
Another practitioner said that by addressing limitations in existing data, the partnership had 
helped expand the reach of educational services to subgroups of children in counties where the 
state had not known that such subgroups existed. Indeed, in at least one case, the partnership 

was initiated by the educational organization to recover lost internal capacity in its research 
department. This district knew what was possible to learn from research and wanted to expand its 
capacity to support the needs of the district through a partnership approach. 
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Conducting and Using Research in the Partnerships 


Given that conducting and using research is a central goal for the RPPs, in this section we 
characterize the general approaches to research the partnerships have used as well as describe 
research use among the practitioners. First, we use our analysis of proposals to characterize the 
research questions, methods, and data sources. Next, we share RPP practitioners’ reports of 
instrumental, conceptual, and symbolic research use. 


Key Findings: 


e The majority of research questions were descriptive or exploratory in nature, although some 
focused on causal relationships or validation of measures or constructs. Most intended to use 
mixed methods and to draw on both new and existing data sources. 


¢ RPP practitioners reported frequently using research for instrumental, conceptual, and symbolic 
purposes. 


¢ Directing resources to programs and redesigning programs were the most common instrumental 
uses of research. 


e« Amajority of RPP participants reported that research had frequently or all of the time “expanded 
(their) understanding of an issue” or “provided a common language and set of ideas.” 


e Over half of the RPP participants reported using research often to “mobilize support for an 
important issue,’ whether frequently or all of the time—a form of symbolic use. 


Research Questions, Methods, and Data Sources 

Each RPP application generally included several research questions that the partnership wanted 
to pursue, for a total of 48 questions in the 27 grant applications we reviewed. Given the focus 

of grants, we expected most questions would be descriptive or relational—that is, they would 
focus on describing current patterns and trends in data related to a focal problem, or explore 
associations between malleable factors and outcomes. Indeed, more than half of the research 
questions we identified were descriptive or exploratory (26 of 48 questions; 54%), and another 8% 
(4 questions) were relational. For example, one partnership focused its work on the descriptive 
question, “What is the average number of annual school changes [transfers], regardless of reason, 
for students in foster care?” Another descriptive question focused on identifying teachers’ 
different blended learning strategies and describing them. Correlational analyses, like exploring 
the relationship between high school performance and college outcomes, are in the category of 
relational questions. 


Partnerships also asked evaluative questions (10; 21%), while some questions (4; 8%) were related 
to causal impact. One evaluative question focused research efforts on assessing the extent to 
which the district’s new assessment system aligned with goals for instructional improvement. An 
example of a causal research question was, “Do performance bonuses improve the retention of 
teachers rated as ‘highly effective’?” Finally, some questions (4; 8%) focused on the validity of an 
instrument or construct. One partnership centered its work on creating and validating teacher 
and student self-report measures for social-emotional learning skills. 


Technical Report No. 2 33 
National Center for Research in Policy and Practice 


The majority of RPP applications (16 of 27) proposed mixed-methods studies that used both 
quantitative and qualitative approaches to investigate their questions. One partnership began with 
a quantitative analysis of EBLs’ outcomes, followed by observations of classroom practices with 
these students. A second project planned to begin with quantitative analyses of administrative 
data sources to create models of student achievement and graduation and pair this work with 
qualitative data analyses of classroom observations, focus group interviews, and surveys to 
identify key leadership and teacher practices. The remaining 11 RPPs planned to use solely 
quantitative research methods. Several projects focused their work on regression analyses of 
variables in combined databases. 


The majority of RPPs (16 of 27) planned to use a combination of new and existing data for their 
research analyses. Six focused on existing data only, and five gathered and analyzed new data only. 
Existing sources included student outcomes data, such as state assessment results; grade point 
averages; English language learner (ELL) proficiency rates; attendance, suspension, expulsion, 
dropout, and graduation data; course pathways data; SAT or ACT results; NAEP, PARCC, 

or Smarter Balanced results; and school mobility metrics. More rarely, existing data sources 
included non-academic outcomes such as measures of foster placement or criminal justice data 
(e.g., recidivism rates). Other data included student, teacher, parent, and school leader survey 
responses; teacher performance or evaluation data; teacher and student demographic data (e.g., 
gender, race/ethnicity, ELL status, free or reduced-price lunch status); and school or program 
characteristics. Newly gathered data often came from focus groups, interviews, instructional logs, 
classroom observations, or case studies. 


Practitioners’ Use of Research 
Next, we consider practitioners’ responses to the questions about research use in decision making. 


Instrumental use. We first asked practitioners whether they were involved in educational 
decisions, such as selecting, designing, or scaling up programs or policies. Table 6 shows the 
percentage of all practitioners reporting involvement in each educational decision. Overall, 
practitioners responding to the survey were most likely to report being involved in designing 
professional development (29; 71%) and least likely to report being involved in purchasing an 
intervention program (18; 44%). 
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Table 6. Percentage of Participating Practitioners and Co-Pls Reporting Involvement in 
Educational Decisions 


Co-Pls in P-20 
research, 
All assessment, and Co-PIls in all 
practitioners All co-Pls accountability other areas 
(n= 43) (n = 20) (n = 10) (n = 10) 
Designing professional development 71% 78% 50% 100% 
Directing resources to a program 68% 83% 63% 100% 
Redesigning a program 66% 78% 50% 100% 
Eliminating a program or policy 63% 67% 50% 80% 
Scaling up a pilot program 62% 74% 56% 90% 
Purchasing an intervention program 44% 61% 50% 70% 


One goal stated in IES’s RFA is that lead co-PIs are in decision-making roles in their educational 
agencies. Indeed, co-PlIs as a group reported greater involvement in these activities when com- 
pared to all practitioners. For instance, 83% of all co-PIs (15 of 18 responding to the item) report- 
ed being involved in directing resources to a program, compared with 68% of all practitioners (28 
of 41 responding to the item). However, involvement by lead co-PIs differed by area of practice 
within their organization. The 10 lead co-PIs in P-20 research, assessment, and accountability of- 
fices reported less involvement in five of the six activities compared to all practitioners, while the 
10 co-PIs in the other areas of practice had higher average involvement for all activities compared 
to all practitioners. 
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Figure 4 below shows the frequency with which survey respondents used research to make 
different types of decisions, for those involved in making those decisions. Approximately three- 
quarters reported using research frequently or all of the time for all but one of the activities listed. 
Practitioners were least likely to use research to eliminate a program or policy. 
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Figure 4. Frequency of instrumental use of research reported by those involved in decision making for each 
activity. 18 < n < 29 practitioners. 
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Conceptual use. The six items that reflected conceptual use asked participants how frequently 
they had encountered research that expanded their understanding of an issue, provided a new 
framework, or brought attention to a new issue (Figure 5). Reports of research serving conceptual 
purposes were fairly frequent, with the overall average falling between “sometimes” and 
“frequently” using research in this way. Over three-quarters of practitioners (32; 76%) reported 
that research had frequently or all of the time “expanded (their) understanding of an issue”; one- 
half (21; 50%) said it “provided a common language and set of ideas.” 
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Figure 5. Frequency of conceptual uses of research reported by all practitioners. 40 < n< 42 practitioners. 
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Symbolic use. The four symbolic use items asked practitioners to report how often they used 
research to convince others or to mobilize support (Figure 6). Close to two-thirds of practitioners 
(26; 62%) reported that they used research frequently or all of the time to “mobilize support for 
an important issue,’ with an equal number of practitioners claiming that they “selectively used 
research because it would support a decision.” 
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Figure 6. Frequency of symbolic uses of research reported by all practitioners. 41 < n < 42 practitioners. 
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Activities of the Partnerships 


It is typical for RPPs to engage in a broad range of activities, given the breadth and ambitions of 
their goals. In this section, we describe survey findings related to the activities of partnerships 
during their work together. 


Key Findings: 


¢ All partnerships are engaged in problem refinement and analysis of data, and more than half are 
doing some design work together, as is common in design research partnerships and networked 
improvement communities. 


e Many partnerships are analyzing outcomes of a policy or program in the partner organization, and 
many are also building large databases to facilitate multiple analyses. 


e Asmall number of RPPs are engaged in Plan-Do-Study-Act cycles, as is typical of networked 
improvement communities. There are, however, large discrepancies between the numbers of 
researchers and practitioners who reported their partnership was involved in such activities. 


All partnerships were engaged in activities focused on refining their understanding of the 
problems they were addressing through analysis and interpreting data. (See Figure 7 below; for 
breakdown by role group, see Appendix D.) Notably, data interpretation was an activity all of the 
partnerships reported doing together. Other activities common to all or nearly all partnerships 
were communicating findings to others in the organization (27 of 27), reaching out to other key 
stakeholders (26 of 27), and communicating findings to the public (26 of 27). Just over two-thirds 
of the RPPs (21 of 27) were examining relationships between “malleable factors” and outcomes, 
and over half (16 of 27) were engaged in activities to relate their findings to other contexts. For 
these two items, there was less agreement within partnerships; in nine partnerships either only 
researchers or practitioners, but not both, reported RPP engagement in these activities. 
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Figure 7. Number of RPPs reporting activities as part of current grant, as reported by at least half of partici- 
pants in each RPP on the survey (n = 27). 


Below, we describe the number of partnerships engaged in activities that are associated with 
different types of partnerships: research alliances, design research partnerships, and networked 
improvement communities (NICs). It is important to note that just because partnerships engaged 
in activities typical of a particular type of partnership, it does not mean they identified as one 

of these types of partnerships. Rather, we use this typology as a way to understand how they are 
more or less like partnerships that we and other researchers have described in the past. 


Partnerships Engaged in Activities Typical of Research Alliances 

Two types of activities that are common in research alliances are the analysis of policies and 
programs in educational organizations and the construction of data archives to facilitate ongoing 
research. Of the 27 partnerships that responded to the survey, 24 reported analyzing the outcomes 
of a policy or program in the organization, and 21 reported building a database that combined 
data sets (Figure 8). Just over half (16 of 27) reported that the partnership was analyzing the 
implementation of a policy or program. 


Reporting rates were similar for researchers and practitioners as to whether these activities 
were occurring. There was less of a shared understanding about whether each partnership was 
analyzing the implementation of policies and programs: For 10 partnerships, either a researcher 
or a practitioner (but not both) said the partnership was involved in the activity. 
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Figure 8. Number of RPPs reporting alliance-type activities as part of current grant, as reported by at least 
half of participants in each RPP on the survey (n=27). 


Partnerships Engaged in Activities Typical of Design Research Partnerships 
Surprisingly, nearly half of all RPPs were already engaged in some design work. For example, 15 
partnerships reported that they had jointly designed a program or intervention that addresses a 
problem for the organization (Figure 9). Twelve RPPs reported testing a co-designed program or 
intervention, and another 12 said they had refined a co-designed program or intervention. 


This finding requires triangulation with evidence from interviews in spring 2017 to interpret 
accurately. It may be that such design work is taking place because partners have been working 
together already and have had time through other grant opportunities to develop interventions 
together. If accurate, it represents the type of activity that the program may expect partnerships 
to be ready to undertake after the conclusion of the grant but not necessarily as a central focus of 
this grant period. 


Nearly three-quarters of the partnerships (20 of 27) had developed conjectures or hypotheses 
about the features needed to address the problems, which might have been expected, given 
the encouragement to partnerships to use the grant funds to identify goals for design and 
development. 
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Figure 9. Number of RPPs engaged in design research-type activities as part of the current grant, as reported 
by at least half of participants in each RPP on the survey (n = 27). 


Partnerships Engaged in Activities Typical of Networked Improvement Communities 
Some partnerships were incorporating strategies related to NICs. For example, about one-third 
(10 of 27) reported engaging in a Plan-Do-Study-Act cycle (Figure 10). This set of activities had 
the greatest discrepancy between researchers and practitioners concerning whether or not the 
partnership was engaged. More practitioners than researchers reported engaging in activities 
related to continuous improvement, using tools from improvement science, and creating a NIC. 
We suspect that our survey questions signaled clearly to researchers—but perhaps not so clearly 
to practitioners—a set of signature methods of improvement science. Practitioners, familiar with 
the popular language of “continuous improvement,’ may have over-reported these activities, so 
the lower reported levels from the PIs may be more accurate. 
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Figure 10. Number of RPPs engaged in NIC-type activities as part of the current grant, as reported by at least 
half of participants in each RPP on the survey (n = 27). 


Technical Report No. 2 43 
National Center for Research in Policy and Practice 


Communication within the Partnerships 


Partnership work requires regular communication between researchers and practitioners. 
Here, we describe survey findings related to the communication strategies that researchers and 
practitioners described in their RPPs. 


Key Findings: 


¢ Participants reported regular and frequent communication with partners, with most members 
engaging in weekly emails or phone calls. 


e Compared with other forms, videoconferences and video meetings were a less frequent means of 
communication. 


Our survey data indicate that partners reported engaging in regular and frequent communication 
with one another (Table 7). Four-fifths (82; 80%) of participants reported talking over the phone 
or texting, either on a weekly or monthly basis. Nearly two-thirds (66; 64%) of participants 
reported exchanging emails on a weekly basis, and another tenth (11; 11%) reported daily emails. 
Members of RPPs also reported that, on at least a monthly or bi-monthly basis, they worked on 
shared documents or online platforms (81; 80%), met in-person (77; 75%), or engaged in video 


meetings (36; 36%). 


Table 7. Percentage of Researchers and Practitioners Using Different Forms of 
Communication within RPPs, by Frequency of Use 


Frequency of Communication 
Never Yearly Bi-monthly Monthly Weekly Daily 


Email 0% 0% 6% 19% 64% 11% 
Phone or text 6% 2% 12% 45% 35% 0% 
Work on shared documents or 11% 10% 21% 33% 22% A% 
online platforms 

In-person meetings 0% 25% 35% 29% 11% 0% 
Teleconference or video meetings 55% 9% 12% 18% 6% 0% 
Other NA 5% 1% 2% 0% 0% 


Note. “Other” types of communication primarily consisted of various types of meetings that RPP members attended 
together outside of those among RPP members only, such as advisory board meetings, meetings with other 
stakeholders, professional learning sessions, or professional conferences (100 < n< 103 researcher and practitioner 


survey respondents). 
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Challenges to the Partnerships 


In this section, we describe findings from the surveys and interviews related to the challenges that 
researchers and practitioners said they faced as they worked together in RPPs. 


Key Findings: 


¢ Big challenges facing partnerships were organizational turnover and obtaining and creating data 
sets. 


e Syncing up schedules of researchers and practitioners and different expectations regarding 
timelines for getting work done were leading challenges identified in the survey, although just one- 
third of interviewees mentioned this challenge. 


Organizational Turnover 

About half of the participants interviewed (26; 46%) raised the issue of organizational turnover 
within the practitioners’ organizations as a particular challenge. This pattern was consistent across 
the interviewed researchers and practitioners. Further, about one-third (38; 37%) of all surveyed 
agreed or strongly agreed that organizational turnover in the educational organization was a 
substantial challenge to their work. (See Figure 11 below; for results by RPP role, see Appendix 
E.) As one participant explained: 


We have had some new people step into some leadership positions. They don't fully 
understand the partnership. Once they understand and are brought on board and 
get a better understanding, we haven't had any long-standing issues. That 

newness of people, that occasionally interjects the need to go back and build 
relationships and understanding. 


This turnover can directly impact a partnership if the people who left were important for the 
partnership's work. One researcher found that she had four different practitioner contacts one 
year, making basic coordination difficult. Another researcher explained that she tried to develop 
multiple relationships as a strategy: “I don’t put all my eggs in one basket of a person there. I get 
to know others, and so when then there’s a departure, I’m not left out in the cold” 


Data Accessibility 

Obtaining and creating data sets for analysis also posed significant challenges, according to 
almost half (24; 44%) of participants interviewed. Some of these challenges were related to 
drafting data-sharing agreements between organizations. One researcher observed, “the data- 
sharing agreement has been the biggest challenge. It’s been taking a really, really, really long 
time.” This was particularly tricky when the team wanted to link identifiable data (e.g., student 
achievement data, teacher evaluation scores) with other variables. Practical challenges emerged 
as teams worked to identify, clean, transfer, and merge data from multiple institutions into one 
database. 
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Timing and Scheduling 

Issues related to timing and scheduling were also frequently cited as challenges. In the survey, 
over half of all respondents (55; 54%) agreed or strongly agreed that differing timelines made 
it difficult to get work done, while almost half (46; 45%) agreed or strongly agreed that the 
schedules of researchers and practitioners were difficult to coordinate. Timing issues were also 
salient during one-third of our interviews (16; 30%). As one practitioner explained: 


It’s a pace thing. I keep a foot in the university world, and I have a foot in the district. 
The district, compared to the university, is so fast-paced. Well, a school is so much 
faster-paced than a district. The patience you need for research, it’s not particularly 
satisfying for educators. It’s not particularly satisfying for policymakers, either. They 
want results now, and they’re not real patient. 


A researcher from another partnership expressed a similar sentiment: 


It is very discouraging to practitioners in general because that time frame is not suitable 
to the immediacy of the question they just posed. We knock on their door, “Let’s get 
together, and come up with questions that are really, really pressing for you right 

now.’ Then we have to issue the caveat, “Well, sorry, but we won't be able to 

act upon that within the next couple, five, or six years.” That defeats the purpose of 
addressing a question of immediate need. 


This reflection on the slower pace of research and faster pace of practitioners’ needs is well 
documented in the literature.*” Members of RPPs tried to address this challenge by developing 
understandings of different timelines and adjusting practices for both researchers and 
practitioners. 


Other Challenges 

Other challenges documented in the research literature on RPPs emerged in our interviews as 
well, including differing priorities (22; 41%), bureaucratic obstacles (19; 35%), lack of practitioner 
understanding of research methods (15; 28%), and not enough time to meet RPP goals (15; 28%). 
Less common challenges were communication difficulties posed by working across the research- 
practice divide (13; 24%) or researchers being too theory-driven (5; 9%). 
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Figure 11. Participants’ agreement that they have faced particular challenges in partnerships. 
Items are negatively worded, and therefore disagreement is an indication that the challenge is infrequent 
(101 < n< 103 researcher and practitioner survey respondents). 
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Perceptions of the Partnerships 


As a descriptive rather than evaluative study, we cannot make inferences about the impact of the 
partnerships. Rather, in this section, we share participants’ self-reported attitudes towards RPP 
work and the perceived benefits of working in these partnerships that interviewees named. 


Key Findings: 
e Participants highly valued working in their partnerships, with all survey participants agreeing or 


strongly agreeing that they would participate in another RPP in the future. 


e Practitioners named several benefits of the work, including a shift in organizational culture 
for research use and increased access to resources and expertise aimed at understanding and 
addressing specific problems of practice. Both researchers and practitioners felt that the quality 
and applicability of research had increased as a result of the partnership. 


Attitudes Toward the Partnerships 

Participants overall had very positive attitudes toward their partnership experiences. As 
shown in Figure 12, all who responded to the survey either agreed or strongly agreed that they 
would participate in a RPP in the future. Nearly all (98; 96%) reported that RPPs are worth 
the investment from IES, and almost all (89; 87%) thought that researcher and practitioner 
participation was equally valued by members of the RPP. Almost everyone surveyed (97; 95%) 
disagreed that the RPP was not worth the time they invested. (For full results by RPP role, see 
Appendix EF.) 
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Figure 12. Participants’ attitudes toward the partnership. Starred (*) items are negative statements 
(disagreement on these items indicates positive attitudes toward the partnership) (100 < n< 102 researcher 
and practitioner survey respondents). 
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Perceived Benefits of Working in Partnership 

In interviews, representatives from 24 of the 27 partnerships noted that the work of the RPP 
differed from their previous experiences either conducting or participating in research studies. 
Both researchers and educators pointed to the benefits of working in an RPP and the need for 
more research to be conducted in this manner. Five themes emerged from this analysis. They 
described their work as beneficial because it was: 


¢ Collaborative, where researchers and practitioners were mutually committed to 
investing significant amounts of time working together on shared goals; 


¢ Focused on problems of practice that were priorities for practitioners; 


¢ Useful to both local actors and the field, in that it was applicable to thinking about 
and solving local problems and could inform researchers and practitioners elsewhere; 


e Guided by research, employing rigorous methods that incorporated the local 
knowledge and expertise of practitioners; and 


¢ Helpful for other aspects of practitioners’ and researchers’ work. 


Participants appreciated the collaborative nature of work in their RPPs. As noted above, this 
collaborative nature meant that the perspectives of both parties contributed to defining the work. 
This may have supported joint ownership of the project goals and mutual learning. Explained one 
practitioner: 


As a large, urban district, we have many, many requests for research from outside 
entities on an annual basis....We have some good relationships with other researchers, 
particularly a couple other local ones. But, it’s not the same collaboration that we would 
see, certainly, with this partnership. Here, we come up with a common research agenda, 
we agree on the particular focus we're going to have, how we're going to conduct 

the research. That’s been the real strength of the partnership. 


For several researchers, the inclusion of practitioners’ voices and their active participation 
improved the quality and applicability of the research. One researcher explained that the close 
involvement of the practice partners meant “a richer set of data than anticipated. It's been slow 
going because there is way more complexity than I expected....They [the practice partners] are 
really digging down deep.’ For practitioners, researchers brought areas of expertise that helped 
with their own understanding of the issues that the partnership set out to address. 


A second benefit named by participants was that the partnerships focused on problems of practice 
that were a priority for practitioners. As one researcher explained: 


[Research studies are usually] a very researcher-driven kind of endeavor. We write 
the proposal. We do all the analytic work. The subsequent follow-up is on our side, 
where we disseminate it to the journals. The extent to which they [the practitioners] 
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implement the findings in practice is secondary to us. This grant really shifted that. It put 
the practitioner in the driver’s seat, saying, “What is it that you need an answer 
to?” 


This shift to starting with a practitioner need was something some educational leaders said 
contributed to the research being useful and applicable to their daily work. For example, one 
researcher shared his experience in developing a set of indicators to identify high school students 
who would benefit from a particular intervention. The district worked with the research team to 
appropriately identify students and provide them with the additional supports. The work was, 

in one researcher's words, “directly useful [because] it came from the district [saying], “We have 

a need. Can you provide us data that can help us with something very specific in terms of an 
intervention?” 


Partnership activities were useful both to local actors and the field, according to the partners we 
interviewed. In addition to directly informing local work, participants mentioned that the scope 
and quality of the work they had done made it relevant to others outside the partnership, citing 
policymakers, stakeholders in their community, and audiences beyond their local context. As one 
PI stated, “We'll learn through this process...and gain buy-in from other institutions across the 
state, and gain champions for some of the innovations that we hope that the state can then scale 
through policy.” 


Another researcher commented that this mix of being applicable at a local level but also impactful 
to outside audiences was one of the reasons that partnership work was so rewarding: 


You actually are making a difference, and you actually are doing something that you 
enjoy doing, but it’s not just about doing it for the exercise of doing it. I think that is 
really, really valuable. If we want to try to have some change, some impact on practice 
and policy, it’s the only way to go about doing it. 


Another named benefit was that these partnerships are guided by rigorous research that values 

and incorporates the local knowledge and expertise of practitioners. Some participants found that 
practitioner organizations were making more extensive use of research—that is, they integrated 

it into their daily work—as a direct result of being part of the partnership. For example, one 
practitioner reported an increase in the perceived value of research on district programming. A 
researcher in a different partnership remarked that the questions that the practitioner partner was 
asking revealed an increasing reliance on research: 


Her first questions are [now], “What data? Can you give me some articles? Where can 
I go to look to see how I should establish these cut scores?” I’m watching them use 
research on a daily basis now to make decisions. They did some of that before, but I’m 
really seeing it as a prevalent practice. 


Finally, interviewees believed partnerships could inform other aspects of practitioners’ and 
researchers’ work in positive ways. Some researchers reported that their partnerships had improved 
the depth of their understanding of the key issues they study. 
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One explained that the work meant he had a “deeper understanding of the policy context, the 
demographics of students, better understanding of the teacher workforce,” which was better than 
“you could ever get doing things at arm's length or using a national data set.’ He believed that the 
partnership enabled him to ask better questions and understand the data better. 


Practitioners also noticed the benefit of using research to inform decisions and how this 
transferred to other aspects of their daily work. As one practitioner explained: 


Whether it’s a curriculum we're planning, or a series of parenting classes, or 
professional development for staff, how important the research is behind that!... 
[The RPP] has really made me understand the value of having a database 

of research for the other work that I’m doing with other groups. 
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Planned Future Activities of the Partnerships 


The current RFA for the IES grant program proposes that partnerships use the two years of grant 
funding to plan for future work together. In this section, we share information regarding the 
reported plans of the participating RPPs for their future partnership work. 


Key Findings: 
e Many RPPs indicated they had plans to apply or had already applied for future grants. 


e Inspite of shared interest in continuing their partnerships, some participants believed this was 
contingent upon securing additional funding. 


Findings from Interviews 

In their interviews, representatives from all RPPs indicated that they would like their researcher- 
practitioner relationships to continue beyond the life of the IES grant. The partnerships were 
pursuing different strategies for continuing their work together. Eight of the 27 with whom we 
spoke had secured no-cost extensions from IES to continue the work for a third year. Twenty of 
the partnerships had either already secured additional funding, were waiting to hear the status of 
grant applications, or were actively pursuing other grants or funding sources. Participants in six 
RPPs stated that even if additional funding was not secured, their partnerships would continue 
informally. 


In a small number of RPPs (3 of 27), there was disagreement among practitioners and researchers 
regarding the certainty of the partnerships’ futures. While the researchers were quite confident 
that they would continue in some capacity, a few practitioners expressed uncertainty. In one RPP, 
a practitioner noted that “it’s pretty premature to have those conversations” since they were only 
one year into their grant. In the other two, issues of capacity and changing leadership outside the 
RPPs were identified as potential reasons the relationships might not endure. 


Future Activities Reported by the Pls 

Looking at the specific activities that participants reported were planned for the future, we see 
two noticeable patterns (Figure 13). First, the PIs on the whole had higher reports of future 
activities than practitioners or researchers overall. (See Appendix G for a breakdown of reported 
future activities by RPP role.) We conjecture that the PIs have more future plans in mind because 
these are researcher-led projects, and therefore they would be most attuned to the next steps for 
the work. Second, the reports for future activities from the Pls (as well as the other groups) were 
relatively low. From the responses, the largest number of PIs (7 of 27) indicated future plans to 
examine malleable factors and to test and adapt a program that the RPP designed, as might be 
expected of a next stage of partnership work. 
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Figure 13. Number of Pls reporting activities planned for after grant concludes, from survey (n = 26). 
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Guidance for IES 


In this section, we summarize practical suggestions for IES in response to a question about how 
the RPP program might be improved. 


Key Findings: 


In the survey, only half of participants reported that a two-year time frame was adequate for RPPs 
to meet their goals, a finding confirmed in the interviews. Interviewees suggested that new RPPs 
may benefit from a three-year grant period to accomplish their aims. 


Partners believed additional funding would be necesssary if grants were extended by one year. A 
longer grant period would require a proportionate increase in funding, participants noted. 


One popular suggestion from participants for addressing the sustainability of partnerships was 
supplementary funds for additional years of work for partnerships that had been successful 
meeting initial goals. 


Participants appreciated opportunities to connect with other IES-funded partnerships at standing 
meetings like the American Educational Research Association or the IES PI meeting. They 
thought that meetings where teams could engage with members of other RPPs would be useful, 
especially if they were focused on common challenges of working in partnerships. 


Two suggestions emerged related to the grant application process. The first was a quicker 
turnaround for application review in keeping with practitioners’ pace of work. The second was for 
IES to encourage and value a wide range of dissemination or communications activities beyond 
peer-reviewed journal articles. 


Length-of-Time Considerations 
Only half of those surveyed (48; 51%) agreed or strongly agreed that two years was an adequate 
amount of time to meet their goals (Figure 14). 
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Figure 14. Participants’ agreement with views regarding the structure of the IES grant program, from survey. 
Starred (*) items are negative statements, such that disagreement on these items indicates positive attitudes 
(95 <n< 100 researcher and practitioner survey respondents). 


Similarly, 30 interviewees (56%) raised the issue of length of the grant, with the majority 
suggesting that current two-year cycle was too short. Several participants pointed out that this 
time frame was a challenge for newer partnerships where the work was just getting off the ground 
and for those with goals that were particularly ambitious. Two PIs explained: 


It needs to be longer than two years. It’s just too short, especially if you're a beginning 
partner, if it’s just at the very beginning, and, for us, if it’s on a broader scale. I know 
I’ve spoken with other folks that are doing similar things, especially in big, urban 
systems. It just takes time. It’s a perfect mechanism to build relationships, but I don’t 
think two years is long enough. 


Two years is a great start for funding in terms of, “How do we get that partnership 
really started? Get the buy in from folks? Get our agenda set?” But, to start seeing 
the realization of that work, actually having an impact on the practice and the policy 
in the region? I think that that’s obviously a more long-term goal. 


Further, this two-year period was challenging because many participants only had one year of 
work together before they needed to write additional proposals to continue. One participant 
compared this to Congressional re-election, explaining: 
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It feels like being elected to Congress because it’s a two-year term, and so at the 
end of your first year, you've got to be campaigning for your next election. Your 
attention is split between what you're doing and how youre going to continue 
the partnership. 


Choosing to wait until the end of the second year to apply for additional funding was also tricky, 
as it created the possibility of a gap year without funding to support the work. One PI described 
this possible scenario: 


You spent two years leading up to pressing questions. The assumption is you had 
some preliminary data, and you really have nailed it. You know exactly what the 
issue is. You have the local data available. There's something really pressing 

at hand. Then, you say, “Now we're going to take a hiatus for a year before we 

know whether we get funding or not. Then we may have to reapply. Hold 

your breath with me for one or two years, and then we'll continue our partnership.” 
That doesn’t work, especially if you spend two years building momentum, and 
then you hit the brakes. 


Several participants noted that the possibility of no-cost extensions on the grants allowed them 
to continue the work past the official two-year mark, a reality that was important for meeting 
project goals. However, these extensions were not officially a part of the grant structure, and not 
all participants assumed they would be awarded. 


The majority of participants suggested that a three-year grant period would be useful with 
(potentially) a fourth year if a no-cost extension was appropriate. Others offered different 
suggestions. One PI suggested a small planning grant in year one, followed by two to five years 
of funding for partnership work, depending on its scope. This is similar to the National Science 
Foundation (NSF) model: 


Consider a similar structure that NSF has, where scaling years are contingent on 
the review that's ongoing within....IES could implement the structure where the 
grant, the partnership grant could be five years. The first two are the initial 

stage of establishing the partnership. If that is accomplished, funding will continue 
as outlined in the proposals for years three, four, and five. 


Funding Considerations 
Funding was another salient issue for participants, and comments on this topic often 
accompanied discussions of grant timelines. 


Current funding levels. Close to two-thirds (64; 65%) of all survey respondents agreed or strongly 
agreed that their partnerships had enough funding to meet their goals. Looking across role 
groups, however, differences emerge. Only half of the PIs (12; 48%) agreed or strongly agreed with 
this statement, compared to three-quarters of practitioners (31; 75%). 
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In the interviews, nine participants (17%) suggested that additional funding for their current 
scope of work would be beneficial. The issue of funding seemed most pressing for RPPs where 
several organizations were involved, such that the funding was split in multiple ways in order 

for each to have some “skin in the game.” One member of a multi-organizational partnership 
said, “There needs to be more attention given to the sustaining of the backbone for these efforts, 
especially when it’s a multi-partner collective impact model, rather than a one-to-one or a one- 
to-one-to-one kind of model” In contrast, members of six RPPs reported that their funding levels 
were sufficient for their current plans of work. One grantee noted that the current funding was 
“an appropriate investment for these initial partnerships.” 


If IES lengthened the grant time frame, as suggested above, participants recommended 
proportionate increases in the grant awards. Four representatives from four different partnerships 
explained that the current funding levels would not likely cover an additional year or two of 
work. According to one PI, “Extending that at least a year, if not two, and then proportionately 
increasing the budget, would be appropriate.” 


Future funding. Another issue raised by close to one-third (16; 30%) of interviewees focused on 
funding opportunities to continue the work of the partnership. As noted above, the short time 
frame of the grant meant that partnerships needed to think about additional funding early on. As 
one PI explained: 


It’s really the first year [when] you need to develop the foundation—develop joint 
goals, develop routines, have time to sit down and talk about big picture, what's 
success for us? It’s the second year where people can really start moving in terms of 
data collection, analysis, getting out reports, insights, and papers. Then, it’s a 
two-year grant. By the end of two years, we already have to think about how 

to sustain it. That's a big challenge. 


Several participants encouraged IES to offer a continuation grant for RPPs who had met their 
goals or who had demonstrated promising results. A “Phase 2” grant would enable those 
partnerships that had made substantial progress to continue their work together and pursue 
necessary next steps to positively impact educational outcomes. This was generally envisioned as a 
new competitive grant program or as smaller discretionary funds distributed by program officers. 
One grantee offered this suggestion: 


I could imagine a process whereby they gave you two years and then there was the 
opportunity to reapply....[It would be] a way to give program officers a little 
discretion and...access to a pool of money that would allow them to fund some 
subset of the RPPs when they were approaching the final year. 


Others wanted more information about different funding opportunities for partnership work after 
the grant’s completion, including from organizations like the Spencer Foundation or NSF. There 
was also interest in understanding how the work of their RPPs could be further developed and 
supported by other IES grant programs. Several participants told us that they did not know how 
their partnership work could continue to be supported by IES, or, as one researcher said, if there 
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there was a “natural progression” to other IES-sponsored research projects. It also was not clear 
to all participants if and how the partnership context would be taken into account for future 
grant proposals submitted to IES (i.e., if the partnership context would mean they would receive 
additional points in applications for other kinds of projects). 


Time to Meet with Other Partnerships 

Most survey respondents (88; 89%) agreed or strongly agreed that “more opportunities to connect 
with other IES-funded partnerships would be valuable to our partnership.’ This sentiment 

was echoed in the interviews. Close to one-third of those interviewed (16; 30%) said that 
opportunities to learn from other IES-funded RPPs were helpful for their work. Six interviewees 
reported that past efforts to bring together RPPs were useful. These events included a convening 
at AERA, attendance at the IES PI meeting, and webinars organized by IES. Said one practitioner, 
“ve attended a couple of the different national meetings for the principal investigators to get 
together, and those have always been very positive in my experience.’ These opportunities 
reportedly gave participants an opportunity to network and learn about new research findings. 


Another theme that emerged from the interviews was a request for dedicated time within the 
larger meetings or a standalone meeting for RPP grantees only. One practitioner explained: 


We were hoping for a meeting of the research partnership grant recipients—our 
own meeting, and that never happened. If it was supposed to, we weren't aware of it. 
We really thought we were gonna go [to the IES PI meeting] with some special 
attention paid to that, and we didn't feel like that happened. 


While the larger meetings were seen as useful for learning about new research findings, some 
participants wanted opportunities to problem-solve and learn from each other about the specific 
challenges that had arisen from working within a partnership setting. Interviewees wanted to 
“talk shop” with others about practical issues like disseminating findings to a range of different 
audiences, learning about other grant opportunities, or identifying strategies to navigate 
organizational turnover. 


Grant Application and Dissemination 
Participants also shared suggestions related to the grant application itself, as well as to the 
dissemination goals of the program. 


Grant application timelines. About one-fifth of interviewees (11; 20%) offered suggestions 

related to the grant application and the application process. For example, they suggested that 

the application timeline and proposal itself could better reflect practitioners’ calendars and pace 
of work. One PI argued, “It takes so long for that IES grant to go through the pipeline,” while 

a different PI suggested there should be a “quicker turnaround” in the application submission, 
review, and question periods. The fact that there was close to a one-year delay between 
application submission and notification was a challenge for some, given that they wanted to focus 
on current, pressing problems of practice. Two practitioners from school districts also suggested 
that IES reconsider the grant application deadline. Early August coincides with the beginning of 
many school district and academic calendars, making it a busy time of year. 
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Dissemination and communication priorities. Finally, some interviewees (8, 15%) of those 
interviewed shared suggestions about dissemination and communication priorities for their 
grants. Most of them noted that peer-reviewed journal articles should not be the only or main 
goal for dissemination. Participants encouraged IES to include a broader set of dissemination 
products or strategies in the RFA as well as in determining overall grant success. One practitioner 
suggested that timely briefings of RPP findings to important stakeholder groups was a key avenue 
for dissemination: 


Do you know how long it’s been since I’ve read a journal article since I've been an 
administrator? The results of any of these projects, there’s some key stakeholders 
that should be hearing about these results. That’s how we base our general budgets. 
If you brought me something super valuable in the way of a program or curriculum 
through a research project, and it was successful, that will change the way we will 
target our funding for the coming year. 


Other suggestions for communication included a social media strategy, one- to two-page 
summaries, and videos. 


Other Suggestions 

A few other recommendations for IES emerged from the interviews, named by one or two 
interviewees each. Two recommended that IES offer technical assistance for data-sharing issues. 
Two participants wanted to be able to connect partnerships with similar content areas, so that 
they could learn from one another. Two individuals requested that IES continue to publicly 
promote the RPP program; in some circles, they explained, the work of RPPs was not valued as 
much as other kinds of research, and IES’s continued public support helped shift that mentality. 
One researcher suggested that IES create a mentor program for interested researchers to be 
matched up with researchers currently involved in partnership work, and a different researcher 
suggested that IES include in the review panels individuals who have engaged in partnership 
work, rather than just traditional researchers. 
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Key Conclusions and Open Questions 


Overall, researchers and practitioners were both positive about the IES RPP program and their 
experiences with it. They reported making significant progress toward multiple objectives, and 
they appreciated the opportunities that the program provided them to build a foundation for 
ongoing joint work. Nearly all individuals who responded to the survey hoped to continue their 
partnership work in the future; many were developing concrete plans and proposals to do so. 
Participants in the study suggested that their projects had increased access to resources and 
expertise aimed at understanding and addressing specific problems of practice. 


We suspect that there is some bias in these positive evaluations, as there is with any self-report. 
Two sources of potential bias are noteworthy because they relate directly to characteristics of the 
partnerships and the program itself. First, we suspect that participants may be overwhelmingly 
positive about their joint work because these were not new relationships between researchers 

and practitioners; all of the partnerships had at least some history of working together. Thus, a 
decision to apply to the program in the first place likely signaled a joint commitment to working 
together. Second, the partnerships are in a precarious position vis-a-vis external funding. It makes 
sense that partners would be quick to express gratitude to IES for the program, not only because 
they provided funding but also because they are a possible funding source in the future. 


One robust conclusion is that the program is attracting partnerships with some history of 
working together. While the RFA notes that the program supports both new and established 
partnerships, there appears to be a tendency toward funding partnerships with some formal 

or informal prior relationship. To investigate that possibility further, it would be worthwhile to 
examine proposals that were not funded and to interview people who considered applying but did 
not, to assess whether this was a consideration. As such, the program may want to investigate the 
following questions: 


Is the program intended to seed partnerships from the very beginning, or is some 
kind of prior relationship or history of work together beneficial and appropriate? 


How might IES consider a partnership’ history when considering the appropriateness 
of its goals and the length of time needed to meet them? 


Our analysis also indicated that over three-quarters (79%) of the PIs were mid-career or senior 
scholars in education research. Because of tenure and promotion requirements, some institutions 
have strong disincentives in place for pre-tenure faculty to become involved in partnership work, 
as this time-consuming work may be viewed as service rather than research. It is difficult to build 
the capacity of the field for partnership activity without the involvement of early career scholars 
so that they can develop more robust and varied skills to carry out effective research in this area. 
Thus, an open question for the program is: 


What support or encouragement might IES provide to early career scholars to 
develop their skills in forming and sustaining RPPs and in conducting rigorous 
research within them? 
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We also found that participants were pursuing a range of goals and activities that extended 
beyond those clearly articulated in the RFA. The partnerships had much larger ambitions to 
impact local policies and practices as well as student outcomes. ‘They also intended to inform 
the work of others beyond their partnerships, including contributing to the knowledge base in 
education. To that end, they engaged in activities to design and test interventions, even in the 
short period of their partnership grants. (The RFA does state that partnerships may articulate a 
plan for future research that includes the development of interventions.) Open questions for the 
RPP program to consider are: 


What range of short- and long-term goals might partnerships be expected to 
specify at the proposal stage? 


What activities related to longer-term goals might be appropriate to encourage, as well? 


As earlier research on RPPs has found, these types of partnerships struggle to achieve synchrony, 
that is, a state in which researchers and practitioners operate at the same time scale so as to 
coordinate activities effectively. It may be hard for researchers to keep up with the “speed of 
practice,” and researchers’ careful analysis proceeds more slowly than is useful to practitioners. 
The asynchrony—a challenge named in the literature for partnerships of all kinds—is exacerbated 
by the need for partnerships to start planning their next proposals when the work has just begun. 
An open question we hope the RPP program will take up is: 


How might IES adapt grant application deadlines and funding time frames to 
enable RPPs to develop efficient patterns for coordinating work before they 
need to develop proposals for continued work? 
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Next Steps in the Study 


Our study will continue over the coming year as we investigate the IES RPP program as part 

of our activities in the National Center for Research in Policy and Practice. Our next steps will 
begin with additional analyses of our Phase 1 data. From our full set of interviews, we will focus 
on the “origin stories” of the partnerships, understanding how and why they came to be. We 

will refine our analysis of partnership goals, determining how RPPs decided upon those goals, 
what informed their selection, and how research was involved. We plan to analyze the individual 
characteristics and organizational conditions that supported partnership development. We are 
also interested in understanding if there are different models for partnering across RPPs. For 
instance, there may be sets of activities that cluster together or different configurations of the 
practitioner-researcher teams. 


According to our roster, almost half (48%) of the practitioners in co-PI roles worked in research, 
assessment, and evaluation offices, and we plan to explore this finding further. In some ways, 

the research office is a natural partner for external researchers. However, this may be a cause for 
concern if educational leaders with decision-making authority over particular policy areas are not 
centrally involved in partnerships. In fact, most co-PIs were involved in key decisions in which 
research could figure prominently, but this was less likely for co-PIs in research, assessment, and 
accountability positions. Our research has not yet documented well how research produced by 
RPPs might be taken up and whether those with authority for relevant decisions are regularly 
involved. Therefore, we need to understand better the kinds of decisions where application of 
research may be relevant as well as who is at the table for decision making in those contexts. 


Phase 2 of this project will begin in spring 2017, and it will involve a second round of interviews 
and surveys with the same sample of partnerships. We plan to adapt our current instruments as 
needed, based on our preliminary findings and additional IES feedback. Our final report will 
focus on the work of the partnerships over time, considering any differences that emerge as they 
mature. This may include shifts in the roles of researchers and practitioners, the goals of partner 
work, and the perceived value of the partnerships. We will analyze items related to organizational 
culture for research use and the research use items to create a portrait of how the educational 
organizations capacity to use research has shifted (or not) based on their participation in RPPs. 
We will also assess if researchers have shifted how they do research as result of their partnership 
work. 
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Appendix A 
Number and Percentage of Respondents and Non-Respondents, by Position and Role in RPP 
Responding Not responding 
Original All Other All Other 
roster researchers Pls researchers | researchers Pls researchers 
Bnaeecene 41 34 18 16 7 1 6 
(50%) (56%) (69%) (46%) (33%) (50%) (32%) 
Researcher a oe 6 19 8 : i 
(33%) (31%) (23%) (37%) (38%) (0%) (42%) 
Center director 7 : 7 2 : 1 2 
(9%) (7%) (8%) (6%) (14%) (50%) (11%) 
Doctoral student a - 0 ; 2 : 3 
(9%) (7%) (0%) (11%) (14%) (0%) (16%) 
a Co-Pls ones ai Co-Pls taka 
practitioners practitioners | practitioners practitioners 
este 26 19 1 8 ? 1 6 
aera ntability (33%) (42%) (52%) (33%) (21%) (25%) (21%) 
Postsecondary 12 8 2 6 4 1 3 
policy and practice (15%) (18%) (10%) (25%) (12%) (25%) (10%) 
Early childhood 12 5 2 3 7 0) 7 
policy and practice (15%) (11%) (10%) (13%) (21%) (0%) (24%) 
K-12 special 7 5 1 4 2 0) 2 
education (9%) (11%) (5%) (17%) (6%) (0%) (7%) 
K-12 federal 7 5 2 3 2 0) 2 
programs (9%) (11%) (10%) (13%) (6%) (0%) (7%) 
K-12 educator 5 2 2 (0) 3 0 3 
evaluation (6%) (4%) (10%) (0%) (9%) (0%) (10%) 
K-12 curriculum 5 0) (0) (0) 5 1 4 
and instruction (6%) (0%) (0%) (0%) (15%) (25%) (14%) 
ate 2 : : 8 2 1 2 
ie} Lo} fe} 0, 0, fey fo} 
and chief officers (5%) (27) (5%) (0%) (9%) (25%) (7%) 
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Appendix B 


Sample Sizes and Response Rates by Position, Area of Practice, and Senior-Level Designation 


Rostered Respondents Non respondents 
Senior-level Senior-level 
Senior-level (response (non-response 
All (% of all) All rate) All rate) 
Professor 41 sar ot 22 ¢ a6 
(68%) (83%) (79%) (17%) (21%) 
Researcher 27 oa id 10 8 _ 
(52%) (70%) (71%) (30%) (29%) 
. 6 4 4 3 2 
Genny altector ’ (86%) (57%) (67%) (43%) (33%) 
0 4 0 3 fe) 
Doctoral student 7 (0%) (57%) (NA) (43%) (NA) 
Principal investigators 28 22 26 a0 2 2 
(79%) (93%) (91%) (7%) (9%) 
26 35 16 19 10 
Other researchers 54 (48%) (65%) (62%) (35%) (38%) 
All researchers 32 48 61 36 21 12 
(including Pls) (59%) (74%) (75%) (26%) (25%) 
K-12 deputy 4 1 1 3 3 
superintendents 4 - 2 6 é i 
sackehier oracer (100%) (25%) (25%) (75%) (75%) 
K-12 curriculum and 5 2 (0) 0 5 2 
instruction (40%) (0%) (0%) (100%) (100%) 
; : 4 5 2 2 2 
K-12 special education 7 (57%) (71%) (50%) (29%) (50%) 
K-12 federal 7 4 5 3 2 1 
programs (57%) (71%) (75%) (29%) (25%) 
K-12 educator 5 4 2 2 3 2 
evaluation (80%) (40%) (50%) (60%) (50%) 
P-20 research, 16 19 13 7 3 
assessment, and 26 ; ‘ 3 rs i 
accountability (62%) (73%) (81%) (27%) (19%) 
Early childhood 12 7 5 3 7 4 
policy and practice (58%) (42%) (43%) (58%) (57%) 
Postsecondary policy 12 9 8 6 4 3 
and practice (75%) (67%) (67%) (33%) (33%) 
Co-principal 25 20 2h 16 4 4 
investigators (80%) (84%) (80%) (16%) (20%) 
sis 30 24 14 29 16 
Other practitioners 53 (57%) (45%) (47%) (55%) (53%) 
All practitioners 78 50 45 30 33 20 
(including co-Pls) (64%) (58%) (60%) (42%) (40%) 
98 106 66 54 32 
Overall a8? (61%) (66%) (67%) (34%) (33%) 
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Appendix C 
Number of RPPs Reporting Specific Goals, by Role of Survey Respondent in RPP (Survey Data) 


Reported as goal of partnership 


At least one 
At least researcher 
50% of RPP and one Researcher Practitioner 
participants PI Co-PI practitioner only only 
n(RPPs) 27 26 21 25 25 25 
Cultivate partnership relationships 
Build a foundation or infrastructure for 27 2% 18 23 2 0 
future work together 
Increase capacity of researchers and 
practitioners to conduct partnership 
work 
Develop the capacity of researchers to 
work in partnership with educational 25 24 19 23 0 2 


leaders 


Develop shared understandings of how 
researchers and practitioners can learn 27 26 18 25 0 0 
from each other 


Conduct and use rigorous research 


Develop the capacity of the organization/ 2% 24 17 22 2 1 
district to use research 

Make progress toward addressing the fo- 

cal problems through research activities 27 26 18 25 0 0 
(e.g., researching an existing intervention) 


Develop research evidence that meets 
standards for publication in a peer-re- 25 19 14 19 3 2 
viewed journal 


Impact local improvement efforts 


Develop a deep understanding of the 


problem the partnership set out to 27 26 18 24 1 0 
investigate 

Improve students’ academic 2% 20 16 18 4 3 
outcomes 

Identify a specific strategy for 24 16 17 18 2 4 
improvement 

Develop approaches to evaluate an 23 16 15 18 2 4 
improvement strategy 

Improve organizational/district processes 

(e.g., changing how students are identified 22 18 13 17 5 3 


for interventions) in order to impact 
students’ outcomes 


Improve organizational/district policies 
(e.g., teacher evaluation) in order to 19 16 10 12 7 5 
impact students’ outcomes 


Improve students’ social-emotional or 


sf 16 14. 10 11 4 6 
non-cognitive outcomes 
Inform the work of others 
Develop research findings that apply 
to other educational organizations or 25 24 15 21 3 a 


districts 
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Appendix D 


Number of RPPs Reporting Specific Activities the Partnership Has Done or Plans to Do, by Role of Survey 
Respondent in RPP (Survey Data) 


Has done or plans to do as part of grant 


At least one 
At least researcher 
50% of RPP and one Researcher Practitioner 
participants PI Co-PI practitioner only only 
n(RPPs) 27 26 20 25 25 25 
Cross-type activities 
Interpret data together 27 26 20 24 1 0 
Communicate findings to others in the 27 25 19 22 3 0 
organization 
Reach out to key stakeholders in the 2% 24 18 22 2 1 
organization 
Communicate findings to the public 26 24 19 20 3 2 
Identify gaps in student outcomes 23 19 16 19 3 1 
Examine relationships between things 21 14 12 15 5 4 
you can change and outcomes 
Connect findings to other contexts 16 12 13 12 5 4 
Alliance-type activities 
Analyze outcomes of a policy or program 24 19 13 16 6 2 
in the organization 
Build database that combines data sets 21 18 14 18 2 af 
Analyze implementation of a policy or 16 412 10 12 5 5 
program in the organization/district 
NIC-type activities 
Conduct Plan-Do-Study-Act cycles to 10 8 6 4 5 5 
address a problem for the organization 
Develop ways for practitioners to quickly 9 5 1 7 1 6 
test whether changes are working 
Use tools from improvement science to 7 4 9 3 5 10 
represent understanding of a problem 
Create a “networked improvement com- 6 5 10 5 4 8 


munity” interested in testing solutions 


Design-type activities 


Develop a set of conjectures or hypothe- 
ses about features of programs needed to 20 17 16 18 3 3 
address a problem 


Jointly design a program or intervention 
that addresses a problem for the 15 14 10 42 3 1 
organization 


Try out a program or intervention the 


partnership designed 12 8 12 9 (0) 7 


Adapt or refine a program the partner- 
ship designed, based on testing it in the 12 8 8 7 3 5 
organization 
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Appendix E 


Percentage of Researchers Reporting Challenges in RPPs, by RPP Role (Survey Data) 


Pls Other researchers 


n SD D A SA n SD D A SA 


Researchers have multiple ways of 
communicating research findings with 25 0% 4% 68% 28%] 35 3% 11% 54% 31% 
practitioners. 


Practitioners have a good understanding of 


24 4% 8% 71% 17%| 35 3% 34% 57% 6% 
research methods. 


* Researchers and practitioners have different 


ear : ; 25 0% 24% 52% 24%| 35 6% 37% 37% 20% 
timelines for getting things done. 


* Researchers’ and practitioners’ schedules 


: 25 16% 40% 44% 0% | 35 6% 34% 54% 6% 
are hard to coordinate. 


* Turnover in the educational organization/ 


une : 25 20% 40% 28% 12%] 35 20% 43% 26% 11% 
district presents substantial challenges. 


* Researchers and practitioners have different 
goals and priorities, and that makes it difficult 25 28% 56% 16% 0% | 35 23% 60% 17% 0% 
for them to work together. 


* There is not enough time for researchers 
and practitioners to work together to make 25 20% 64% 16% 0% | 35 14% 69% 17% 0% 
progress. 


* Researchers are driven by theory, not the 
problems facing educational organization/ 25 72% 24% 4% 0% | 35 40% 54% 6% 0% 
districts. 


Note. Starred items are negative statements (disagreement on these items indicates that these were not strongly sup- 
ported as challenges); SD = Strongly Disagree; D = Disagree; A = Agree; SA = Strongly Agree. 
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Percentage of Practitioners Reporting Challenges in RPPs, by RPP Role (Survey Data) 


Co-Pls Other practitioners 


n SD D A SA n SD D A SA 


Researchers have multiple ways of 
communicating research findings with 19 0% 5% 84% 11%|] 23 O% 4% 78% 17% 
practitioners. 


Practitioners have a good understanding of 


19 0% 21% 58% 21%} 23 0% 39% 57% 4% 
research methods. 


* Researchers and practitioners have different 


nie : ; 19 16% 42% 37% 5% | 23 13% 52% 22% 13% 
timelines for getting things done. 


* Researchers’ and practitioners’ schedules 


: 19 16% 63% 21% 0% | 23 13% 43% 43% 0% 
are hard to coordinate. 


* Turnover in the educational organization/ 


9 fe) 9, 9 9 9 9 0, 
district presents substantial challenges. ae: ie ile. Oi, ae ee, ee Rade Oe Ne 


* Researchers and practitioners have different 
goals and priorities, and that makes it difficult 20 45% 40% 15% 0% | 23 22% 48% 26% 4% 
for them to work together. 


* There is not enough time for researchers 
and practitioners to work together to make 19 16% 74% 11% O% | 23 17% 61% 22% 0% 
progress. 


* Researchers are driven by theory, not the 
problems facing educational organization/ 19 37% 53% 11% O% | 23 26% 61% 13% 0% 
districts. 


Note. Starred items are negative statements; disagreement on these items indicates that these were not strongly 


supported as challenges. SD = Strongly Disagree; D = Disagree; A = Agree; SA = Strongly Agree. 
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Appendix F 


Attitudes Towards Partnership Reported by Researchers on Survey 


Pls Other researchers 


n SD D A SA n SD D A SA 


pwOulel par lcipate ts aheseat shel 25 0% 0% 12% 88%| 35 0% 0% 11% 89% 
practitioner partnership in the future. 


Researcher-practitioner partnerships are 25 A% 0% 12% 84% | 35 0% 3% 23% 74% 
worth the investment from IES. 


Researcher and practitioner participation iS 25 0% 8% 32% 60% | 35 0% 14% 31% 54% 
equally valued by members of the partnership. 


*1 would not recommend to a colleague that 
they join or form a researcher-practitioner 25 72% 20% 4% 4% | 35 80% 14% 3% 3% 


partnership. 


* IES should prioritize funding causal studies 
(e.g., randomized control trials) overresearch- 25 40% 52% 4% 4% | 35 49% 43% 6% 3% 


er-practitioner partnerships. 


The researcher-practitioner partnership 25 88% 8% 0% 4% | 35 83% 14% 3% 0% 
was not worth the time we invested in it. 


Note. Starred items are negative statements; disagreement on these items indicates positive attitudes towards the 
partnership; SD = Strongly Disagree; D = Disagree; A = Agree; SA = Strongly Agree. 
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Practitioners’ Agreement with Attitude Statements About Partnerships, by RPP Role 


(Survey Data) 
Co-Pls Other practitioners 
n SD D A SA n SD D A SA 
| would participate in a researcher-practi- 19 0% 0% 16% 84%! 23 0% 0% 22% 78% 


tioner partnership in the future. 


Researcher-practitioner partnerships are 419 5% 0% 11% 84% | 23 0% A% 22% 74% 
worth the investment from IES. 


Researcher and practitioner participation is 419 5% 5% 32% 58% | 23 0% 17% 26% 57% 
equally valued by members of the partnership. 


*1| would not recommend to a colleague that 
they join or form a researcher-practitioner 19 68% 16% 5% 11%] 23 65% 22% 4% 9% 


partnership. 


* IES should prioritize funding causal studies 
(e.g., randomized control trials) overresearch- 19 21% 74% 0% 5% | 21 438% 48% 5% 5% 


er-practitioner partnerships. 


The researcher practitioner partnership 19 79% 16% 5% 0% | 23 65% 26% 0% 9% 
was not worth the time we invested in it. 


Note. Starred items are negative statements; disagreement on these items indicates positive attitudes towards the 
partnership. SD = Strongly Disagree; D = Disagree; A = Agree; SA = Strongly Agree. 
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Appendix G 


How Different Groups Reported Activities that the Partnership Plans to Do After the Grant on 
Survey, by Number of RPPs 


Plans to do after the grant 


At least one 
At least researcher 
50% of RPP and one Researcher Practitioner 
participants PI Co-PI practitioner only only 
n(RPPs) 27 26 20 25 25 25 
Cross-type activities 
Interpret data together 0 0 0 0 (0) 0 
Communicate findings to others in the 0 0 1 0 1 1 
organization 
Reach out to key stakeholders in the 0 1 1 0 1 1 
organization 
Communicate findings to the public 2 1 1 dl 3 4 
Identify gaps in student outcomes 0 2 1 O 3 uf 
Examine relationships between things 2 7 4 i 6 3 
you can change and outcomes 
Connect findings to other contexts 2 2 2 0 6 5 
Alliance-type activities 
Analyze outcomes of a policy or program 3 5 2 2 3 2 
in the organization 
Build database that combines data sets 0 il 2 0 1 2 
Analyze implementation of a policy or 3 4 3 2 4 2 
program in the organization/district 
NIC-type activities 
Conduct Plan-Do-Study-Act cycles to 1 2 0 0 3 0 
address a problem for the organization 
Develop ways for practitioners to quickly 1 5 0 0 6 0 
test whether changes are working 
Use tools from improvement science to 1 3 0 0 3 1 
represent understanding of a problem 
Create a “networked improvement com- 1 3 1 0 3 1 


munity” interested in testing solutions 


Design-type activities 


Develop a set of conjectures or hypothe- 
ses about features of programs needed to 0 1 0 O 5 0 
address a problem 


Jointly design a program or intervention 
that addresses a problem for the 2 4 2 1 5 3 
organization 


Try out a program or intervention the 
partnership designed 


Adapt or refine a program the partner- 
ship designed, based on testing it in the 3 6 1 0 9 il 
organization 
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